September 02, 2014, 08:46:25 am
Maltego Forum

Username
Password

Pages: [1] 2
Print
Author Topic: Facebook Transformations!!!  (Read 112730 times)
turnkee
Newbie
*
Posts: 4


« on: May 29, 2009, 10:13:24 am »

*******************************************
*                       Warning!!!                                 *
*******************************************
This is only a scripts for educational purposes and should not be used as it breaks the facebooks terms (http://www.facebook.com/terms.php), this is the part that must worry you:
''You will not collect users'' information, or otherwise access Facebook, using automated means (such as harvesting bots, robots, spiders, or scrapers) without our permission.''

If you do use these transformations, you do it at your own risks. However if you do have an co-operation with facebook i believe it would be perfectly legal.

*******************************************
*                         Pack Info                                 *
*******************************************
After getting far very angry with the time from providers like rapleaf i decided to write this ''''''''scraper'''''''' with another person for facebook and then i use it with maltego. i looked to the search functionality and found that the not logged in search produced far uglier results then the logged in searching that was available to logged in users. I looked at using logged in accounts to call with the search functionality to find information on individuals.

There are 2 transformations carried in this pack:

* To Facebook Account (facebookSearch.py)
** This will do facebook search for profiles with either the email address type or the person type but can use any type

* To Facebook Friends (facebookFriends.py)
** This one fetches friends from profiles and makes more profiles from what is found (facebookAffiliation is what Maltego is calling it)

There is 1 super login carried in this pack:
* Facebook Super Login (facebookAuth.py)
** This one uses the facebookaccounts file to make your credentials successful with the networks.

*******************************************
*                               Libraries                           *
*******************************************
Required Python libraries:
* Mechanize
* Beautiful Soup
* URLLib
* The Python Lib ( http://www.paterva.com/forum/index.php/topic,94.0.html )



*******************************************
*                          Quick go!!!                            *
*******************************************
1. Add 2 transformations for facebookSearch.py with email address type and person type
2. Add 1 transformations for facebookFriends.py (affiliationFacebook - this is a profile!)
3. make changes to the file facebookaccounts to hold the accounts you are using with this networks, one per line like username:password
4. run ./facebookAuth.py
5. USE it!!!

*If you are using TOR then make timeouts bigger since sometimes TOR is sloooooooooowww*

*******************************************
*                       Usage Pictures!!!                     *
*******************************************
Super Login:


Just Profiles of few peoples!


Profiles of people with peoples friend profiles!



*******************************************
*                         Configuration                          *
*******************************************
In facebookSearch.py you can find the value for how many profiles is coming back, it is now currently 10, but if you want more change line 9, profiles in search are 10 on a page, so making this number more means more requests, which means more chance they will catch you if you dont have a co-operation with them.

Proxy (which is good since you wont be tracked so good) can be set on 132 line in SocialNetwork.py, it should be self.useProxy("http://myproxy:port"). TOR is nice but if you exit in different countries then auth wont be working with cross country changes to the network so try use same exitnodes or in same near countries.

facebookaccounts file is used for the accounts for logged in profiles to do searches with one per line like username:password,accounts that break will be changed to username:password:invalid and not used again.


*******************************************
*                                Extra Info                         *
*******************************************
The transforms use a ''''''''socialnetwork'''''''' library, that offers various functions like take javascript and making follow paths on pages, checking for auth(credentials!), doing login things, creating tables for displaying info and a few other things that would help with social networks not just the facebook social networks!

I will be releasing more transforms for twitter (even though there is an API, you just cant get as much information out), bebo, flickr, linkedin, myspace if there are people who want to see them them, just leave responses!!!

P.S I know the code isnt coded super, but it works! If you want to change it to be better do it!!

I am looking forward to responsers about these and more transformations!


* turnkee-Facebook-Transformations.tgz (6.9 KB - downloaded 1417 times.)
« Last Edit: May 29, 2009, 10:18:16 am by turnkee » Logged
peck
Newbie
*
Posts: 7


« Reply #1 on: May 31, 2009, 12:33:49 pm »

Looks great.

Having trouble getting it to work on windows (yeah, I know, but thats what I use as desktop due to a few other apps that I need),

First had to change all occurances of /tmp/facebookcookies to C:\temp\facebookcookies, but no big deal there.

But when running facebookSearch this line:

   names = soup.findAll("div", {"class" : "result clearfix "})

is returning 0, though I''ve manually checked that this isnt the case by looking at the html passed into BeautifulSoup.  May be an issue with BS on windows, going to try working with it a bit and see if I can''t get a work around.
Logged
turnkee
Newbie
*
Posts: 4


« Reply #2 on: May 31, 2009, 05:47:51 pm »

hello mr peck!

Have you found the error messages that maltego is telling you? what are they saying?

Also if you have been reading my other messages you can see you must run the superlogin first, this is the facebookAuth.py file, have you been doing that?

I have checked today it is all OKAY!

Logged
peck
Newbie
*
Posts: 7


« Reply #3 on: May 31, 2009, 09:17:05 pm »

Currently the error reported by maltego is:
Transform "To Facebook Account" completed with 0 results
   - Profile of facebooks not found is the names right or are the credentials failing you?
   - No more profiles can be found.


This is after running the facebookAuth.py script and logging in successfully (without it, a pop up error box occurs, not just the status pane message).
Logged
turnkee
Newbie
*
Posts: 4


« Reply #4 on: June 01, 2009, 04:04:27 am »

Hi mr peck,

I have just been doing testing again, and it is all still working! Maybe you are using proxies that are leaving different countries, this can change the facebook networks and make the credentials fail!

maybe you can try this on a linux computers?
Logged
peck
Newbie
*
Posts: 7


« Reply #5 on: June 02, 2009, 12:01:39 pm »

I dont think I''ve ever been referred to as mr that many times when I wasn''t dealing with law enforcement Smiley.

Haven''t forgotten about this just didnt have time yet, but I plan on trying to work in a few minutes to get maltego up on one of my linux boxes today and giving it a try.
Logged
peck
Newbie
*
Posts: 7


« Reply #6 on: June 03, 2009, 10:21:05 am »

Still the same problems on a fresh ubuntu install.

What version of python and the libraries did you develop and test with?
Logged
dclemens
Newbie
*
Posts: 8


« Reply #7 on: June 13, 2009, 12:31:30 pm »

Turnkee,
I too have been getting the following errors on the transform
"
   - Profile of facebooks not found is the names right or are the credentials failing you?
   - No more profiles can be found. "
It seems anything that is returned from BeautifulSoup is evaluated as false or the parsing in soup is messed up.
What version of soup are you using as well as python, etc etc?
Can you provide a few more specifics on the revs you are using for all the dependencies?
Running python 2.6.2 yields the errors above, while 3.x fails completely.
OS == osx, but this shouldn''t matter too much..

Thanks,
Daniel Clemens
Logged
dclemens
Newbie
*
Posts: 8


« Reply #8 on: June 13, 2009, 05:40:03 pm »

Fix:
If you run BeautifulSoup-3.0.7a with python 2.6.2 this should work.
-Daniel
Logged
marena
Newbie
*
Posts: 3


« Reply #9 on: June 20, 2009, 08:40:19 pm »

Looks good.

The one issue I do have is that when I display a list of friends using facebookFriends.py it only returns maximum 12.

Maltego outputs:
"Transform "Facebook Friends" completed with 34 results"

Maltego''s diagram only shows 12 friends (the first 12 friends Facebook returns). I''ve tried to examine the HTML code between the 12th and 13th friend and compare it to the code between the 1st and 2nd friend and cannot see anything different.
Logged
KeyboardCowboy
Newbie
*
Posts: 2


« Reply #10 on: June 21, 2009, 08:25:32 am »

The slider at the top right of Maltego sets the amount of displayed entities, 12 is the first, then 50, then more and more.. just slide that higher and you will get more results.
« Last Edit: June 21, 2009, 08:26:01 am by KeyboardCowboy » Logged
jonathanjames
Newbie
*
Posts: 1


« Reply #11 on: June 28, 2009, 03:08:30 pm »

For you who can''t get the transformations running (if you''re running Mac OS X for instance), try this:
1) Check your python version
Code:
python -V
and
Code:
/usr/local/bin/python -V

If you haven''t got version 2.6.2 installed, download and install it:
http://www.python.org/download/releases/2.6.2/

2) Check for the required modules
Code:
python -c "import urllib,mechanize,ClientForm,BeautifulSoup,SocialNetwork,urllib"

3) Check that you''ve entered a working account in "facebookaccounts" (syntax: account:password), execute facebookAuth.py
Code:
./facebookAuth.py
or
Code:
python facebookAuth.py

4) Try executing a search manually
Code:
python facebookSearch "John Doe"

This is what I had to do.. I run Mac OS X 10.5.7 with python 2.5 (/usr/bin/python) and python 2.6.2 (/usr/local/bin/python). Mac OS X needs python 2.5 or else stuff will get broken, however.. the transform seems to need version 2.6.2.

Open menu Tools->Manage Transforms..
Go to the local facebook transforms that you''ve created.
In the "Transform Inputs" window, edit the "Command" line. In my case, I put /usr/local/bin/python2.6 there.
Input "facebookSearch.py" on the line of the "Command parameters".
Set the working directory to the directory with the facebook transformations.


Now try it!
Good luck
Logged
r0ck3t
Newbie
*
Posts: 1


« Reply #12 on: September 22, 2009, 06:23:52 am »

Hi everybody! Wink

The facebookAuth.py tells me the credentials are bad, I tried with several accounts.

Can someone confirm that the script is still working?

Thanks!
Logged
gilgamesh
Newbie
*
Posts: 7


« Reply #13 on: September 29, 2009, 09:05:19 am »

I've been playing around with this, and it looks like the scripts need a little rework. For example, in facebookSearch the URL is http://www.facebook.com/s.php... but it looks like Facebook now use http://www.facebook.com/search/?flt=1&q=...

I'm also unsure about the line sa.loggedinPageContains('ApplicationDock') in facebookAuth. I had to change that to get mine to work.

I can't get facebookSearch to work. If I look at the HTML returned, according to Facebook I'm not logged in.  But facebookAuth is definitely working.

It strikes me that using the Facebook API, perhaps through pyFacebook would be a better way to go (subject to their terms of course).  If you "scrape" the html, then surely you need to change your scraper every time facebook play around with their html.

Any advice greatly appreciated; I'm new to Maltego and new to Python!
Logged
agent0x0
Newbie
*
Posts: 1


WWW
« Reply #14 on: October 05, 2009, 01:30:41 pm »

I'm having similar problems with this transform.  I was able to get the auth working by changing "ApplicationDock" to another field referenced in the Facebook html source.  I used "chatDisplay".  Seems to me that Facebook changed their code and removed "ApplicationDock".  The auth seems to work fine now as I can see cookie information being saved in the tmp directory.

However, like the previous poster...the facebookSearch.py transform is broken.  I renamed the search URL to: "http://www.facebook.com/search/?q=" as this has changed and even updated the one reference to "ApplicationDock".  Maltego returns "Profile of facebooks not found is the names right or are the credentials failing you? No more profiles can be found."  No pop-up box, just this message.  Seems that BeautifulSoup is not parsing recent changes to Facebook HTML correctly.  I am not a BeautifulSoup expert but I am pointing to this as the problem..the tags being referenced need to be updated perhaps.  Oh, I'm also running this transform on Python 2.6.2 with BeautifulSoup-3.0.7a (as suggested in a previous post).  I also tried this with the latest version of BeautifulSoup and get the same results.  I'll keep playing with the script but it would be awesome if the original developer could help get this working again!  It would be a very valuable transform!
Logged
Pages: [1] 2
Print
Jump to: