New API methods to retrieve social graph without pagination

18 views
Skip to first unread message

Alex Payne

unread,
Feb 3, 2009, 8:01:05 PM2/3/09
to Twitter Development Talk, twitter-ap...@googlegroups.com
Happy to announce two new API methods today, delivered in response to
developer demand for an easier way to keep tabs on users' social graphs.
The methods, /friends/ids and /followers/ids, return the entire list of
numeric user IDs for a user's set of followed and following users,
respectively. Responses to these methods are cached until the user's
social graph changes. The responses come direct from our denormalized
list data stores, and should be reasonably fast even for users with a
large number of followers/follows.

These new methods are most useful for services that are maintaining a
cache of user details. If you see a user ID that you don't have cached,
you'll have to call /users/show to retrieve that user's details. But for
services with large user bases, or those that simply want to diff a
user's social graph over time, we hope these methods will come in handy.

You can find the documentation at
http://apiwiki.twitter.com/REST-API-Documentation#SocialGraphMethods.

--
Alex Payne - API Lead, Twitter, Inc.
http://twitter.com/al3x

Sylvain Munaut

unread,
Feb 3, 2009, 8:20:21 PM2/3/09
to Twitter Development Talk
Great ! Thanks a lot.

Small details, when no credential is provided, the follower method
correctly return 'could not authenticate you' while the friends method
return a HTML error page.

Sylvain

Alex Payne

unread,
Feb 3, 2009, 8:29:24 PM2/3/09
to twitter-deve...@googlegroups.com
Ack, yes, we caught that just after it went out. Currently pushing out a
fix that didn't make it into our earlier deploy.

Sylvain Munaut

unread,
Feb 3, 2009, 8:25:39 PM2/3/09
to Twitter Development Talk
Mmmh, I may have spoken a little too soon ...

* As I mentionned, I must provide a login / password to do the call
* Whatever id/screen_name I ask for, I always get MY list of friend /
follower and not the one I asked.

Is anybody getting this ?


pnoeric

unread,
Feb 3, 2009, 8:40:24 PM2/3/09
to Twitter Development Talk
hey! awesome news. I'm reworking my code to use user IDs so this will
help a lot. thanks much. E

On Feb 3, 5:01 pm, Alex Payne <a...@twitter.com> wrote:
> Happy to announce two new API methods today, delivered in response to
> developer demand for an easier way to keep tabs on users' social graphs.
> The methods, /friends/ids and /followers/ids, return the entire list of
> numeric user IDs for a user's set of followed and following users,
> respectively. Responses to these methods are cached until the user's
> social graph changes. The responses come direct from our denormalized
> list data stores, and should be reasonably fast even for users with a
> large number of followers/follows.
>
> These new methods are most useful for services that are maintaining a
> cache of user details. If you see a user ID that you don't have cached,
> you'll have to call /users/show to retrieve that user's details. But for
> services with large user bases, or those that simply want to diff a
> user's social graph over time, we hope these methods will come in handy.
>
> You can find the documentation athttp://apiwiki.twitter.com/REST-API-Documentation#SocialGraphMethods.

Gilles Frydman

unread,
Feb 3, 2009, 9:51:35 PM2/3/09
to twitter-deve...@googlegroups.com
It is working fine for me.
This is very useful, thanks!

I have a question about created_at. I couldn't find a definition in the documentation.
Are all the created_at data standardized on GMT or does it vary on the user location?
It looks like GMT but I still get some weird results mixed with valid results, based on that assumption.

--
Gilles Frydman
ACOR.org

Jesse Stay

unread,
Feb 3, 2009, 11:34:50 PM2/3/09
to twitter-deve...@googlegroups.com
Well done Alex.  While the 20k requests/hr API limit will still limit the number of users we are able to have as we grow larger down the road, this will make a huge difference for us and let us keep growing for much longer.  Also, I like that this encourages apps not to rely on the username of a person as the unique identifier, as the username can change at any time - I've noticed that most apps are relying on the username and not user id.  Working to get this in place ASAP.

Thanks,

Jesse

Matt K.

unread,
Feb 3, 2009, 11:17:59 PM2/3/09
to Twitter Development Talk
Alex -

This is a great addition to the API - will make things much easier.

Quick question (and I apologize if this is already documented): do the
followers / friends always come in descending order of when they
friendship/follow was created? In other words will the most recent
follow/friend always be first?

I know the original followers call was ordered in the order in which
the follower joined twitter. Hoping this isn't set up the same way -
it would be nice to basically stop iterating over the list once a
repeat friend/follower is found.

Thanks for the clarification,
Matt

On Feb 3, 5:01 pm, Alex Payne <a...@twitter.com> wrote:
> Happy to announce two new API methods today, delivered in response to
> developer demand for an easier way to keep tabs on users' social graphs.
> The methods, /friends/ids and /followers/ids, return the entire list of
> numeric user IDs for a user's set of followed and following users,
> respectively. Responses to these methods are cached until the user's
> social graph changes. The responses come direct from our denormalized
> list data stores, and should be reasonably fast even for users with a
> large number of followers/follows.
>
> These new methods are most useful for services that are maintaining a
> cache of user details. If you see a user ID that you don't have cached,
> you'll have to call /users/show to retrieve that user's details. But for
> services with large user bases, or those that simply want to diff a
> user's social graph over time, we hope these methods will come in handy.
>
> You can find the documentation athttp://apiwiki.twitter.com/REST-API-Documentation#SocialGraphMethods.

dougw

unread,
Feb 3, 2009, 11:45:19 PM2/3/09
to Twitter Development Talk
Just wanted to supply sample output for the curious:

curl --basic --user dougw:PASSWORD http://twitter.com/followers/ids.xml
<?xml version="1.0" encoding="UTF-8"?>
<ids>
<id>2396591</id>
<id>13348</id>
<id>6014732</id>
<id>9300822</id>
</ids>

@dougw

rhysmeister

unread,
Feb 4, 2009, 4:19:06 AM2/4/09
to Twitter Development Talk
A great addition to the API. Thanks for lsitening!

Rhys

On Feb 4, 1:01 am, Alex Payne <a...@twitter.com> wrote:
> Happy to announce two new API methods today, delivered in response to
> developer demand for an easier way to keep tabs on users' social graphs.
> The methods, /friends/ids and /followers/ids, return the entire list of
> numeric user IDs for a user's set of followed and following users,
> respectively. Responses to these methods are cached until the user's
> social graph changes. The responses come direct from our denormalized
> list data stores, and should be reasonably fast even for users with a
> large number of followers/follows.
>
> These new methods are most useful for services that are maintaining a
> cache of user details. If you see a user ID that you don't have cached,
> you'll have to call /users/show to retrieve that user's details. But for
> services with large user bases, or those that simply want to diff a
> user's social graph over time, we hope these methods will come in handy.
>
> You can find the documentation athttp://apiwiki.twitter.com/REST-API-Documentation#SocialGraphMethods.

iematthew

unread,
Feb 4, 2009, 12:11:24 PM2/4/09
to Twitter Development Talk
This is great news. Will make life much easier for a lot of people.
Thanks!

Burhan TANWEER

unread,
Feb 4, 2009, 1:07:33 PM2/4/09
to twitter-deve...@googlegroups.com
Wonderful  news !!!
 
We will be using for our social search engine application. Thanks for coming up with these new API for friends and followers ids. I apprecaite it.

--
Sincerely,

Burhan Tanweer
Founder
www.CorrectSearch.com
Phone:  877-807-1818
sup...@correctsearch.com

Nicole Simon

unread,
Feb 4, 2009, 2:28:06 PM2/4/09
to twitter-deve...@googlegroups.com
Awesome. I still do downloads of follower lists because it is an easier way to get a batch of
information from the profile data than requesting every single one, but this
already is a huge advancement.

Small suggestion: Add a last changed info to it for social graph change or a number?

Like if I have in my db that user X has 200 follower and 200 following, and the new number
is 300 and 210 I might say 'woops, need to check follower but not the following'?

Speaking of dates: Is there any way the new api does not require the funny
dates or is there a method I overlooked not to need to use sth like this:

Tue%2C+27+Mar+2007+22%3A55%3A48+GMT

0703272255 i can easily build as year month day etc, but the additional
name of the month and day complicates things unecessarily.
thanks
Nicole


rhysmeister

unread,
Feb 4, 2009, 5:29:18 PM2/4/09
to Twitter Development Talk
I'm getting duplicates in some of my calls to these methods. I've only
seen it happen in those with a very large number of followers

http://twitter.com/Wossy - 56, 000 followers with 11 duplicates.
http://twitter.com/stephenfry - 120, 000 followers with 164
duplicates.
http://twitter.com/kevinrose - 98, 000 followers with 16 duplicates.


On Feb 4, 1:01 am, Alex Payne <a...@twitter.com> wrote:
> Happy to announce two new API methods today, delivered in response to
> developer demand for an easier way to keep tabs on users' social graphs.
> The methods, /friends/ids and /followers/ids, return the entire list of
> numeric user IDs for a user's set of followed and following users,
> respectively. Responses to these methods are cached until the user's
> social graph changes. The responses come direct from our denormalized
> list data stores, and should be reasonably fast even for users with a
> large number of followers/follows.
>
> These new methods are most useful for services that are maintaining a
> cache of user details. If you see a user ID that you don't have cached,
> you'll have to call /users/show to retrieve that user's details. But for
> services with large user bases, or those that simply want to diff a
> user's social graph over time, we hope these methods will come in handy.
>
> You can find the documentation athttp://apiwiki.twitter.com/REST-API-Documentation#SocialGraphMethods.

Alex Payne

unread,
Feb 4, 2009, 7:31:56 PM2/4/09
to twitter-deve...@googlegroups.com
The response should be ordered with most recent followed/followers first
in the list.

Another developer noted duplicates; we'll look into that.

Burhan TANWEER

unread,
Feb 4, 2009, 9:59:49 PM2/4/09
to twitter-deve...@googlegroups.com

James Deville

unread,
Feb 4, 2009, 10:07:07 PM2/4/09
to twitter-deve...@googlegroups.com
Any chance of a easy way to map this to usernames? We want the friends list for Witty (and I imagine others), but we don't need full profiles, just this + username. This won't help us otherwise since we'll need to map the entire list to usernames, which will require too many requests.

JD

Ninjamonk

unread,
Feb 5, 2009, 3:34:42 AM2/5/09
to Twitter Development Talk
This is really useful, however would be even more useful if you
offered an unrate limited service to return the username for each
userid.

Problems I foresee.
1 user uses your web based app and the user has 10k followers and 10k
friends and thats your limit used up for the hour.

Even with caching using this is pointless for apps with large numbers
of users.

solutions that would work for me.

return the user name as well as userid(yes I know this can change but
most people will want to get it, make it optional)
or
let us use the search api to return user data using a userid like the
username.





On Feb 5, 3:07 am, James Deville <james.devi...@gmail.com> wrote:
> Any chance of a easy way to map this to usernames? We want the friends list
> for Witty (and I imagine others), but we don't need full profiles, just this
> + username. This won't help us otherwise since we'll need to map the entire
> list to usernames, which will require too many requests.
>
> JD
>

Damon Clinkscales

unread,
Feb 5, 2009, 4:01:31 AM2/5/09
to twitter-deve...@googlegroups.com
On 2/5/09, Ninjamonk <dar...@stuartmedia.co.uk> wrote:
> This is really useful, however would be even more useful if you
> offered an unrate limited service to return the username for each
> userid.

> On Feb 5, 3:07 am, James Deville <james.devi...@gmail.com> wrote:


> > Any chance of a easy way to map this to usernames? We want the friends list
> > for Witty (and I imagine others), but we don't need full profiles, just this
> > + username. This won't help us otherwise since we'll need to map the entire
> > list to usernames, which will require too many requests.

I agree that having the username would be useful. If it's not a bunch
of extra strain to include it, that would be splendid. If we have the
username, we can build URLs to Twitter profile pages and/or display
something meaningful (a name) to a user.

thanks,
-damon
--
http://twitter.com/damon

Ryan Pitts

unread,
Feb 5, 2009, 4:34:27 AM2/5/09
to Twitter Development Talk
Very cool - these are exactly the methods I was looking for to do some
diff stuff.

I noticed, though, that an API call fails for a protected user's
friends/followers IDs, even though I can browse to that same person's
friends and followers on the HTML side. Is this on purpose? Would love
to be able to grab these IDs without having to require login by a user
with a relationship to the protected user.


On Feb 3, 5:01 pm, Alex Payne <a...@twitter.com> wrote:
> Happy to announce two new API methods today, delivered in response to
> developer demand for an easier way to keep tabs on users' social graphs.
> The methods, /friends/ids and /followers/ids, return the entire list of
> numeric user IDs for a user's set of followed and following users,
> respectively. Responses to these methods are cached until the user's
> social graph changes. The responses come direct from our denormalized
> list data stores, and should be reasonably fast even for users with a
> large number of followers/follows.
>
> These new methods are most useful for services that are maintaining a
> cache of user details. If you see a user ID that you don't have cached,
> you'll have to call /users/show to retrieve that user's details. But for
> services with large user bases, or those that simply want to diff a
> user's social graph over time, we hope these methods will come in handy.
>
> You can find the documentation athttp://apiwiki.twitter.com/REST-API-Documentation#SocialGraphMethods.

jstrellner

unread,
Feb 5, 2009, 11:40:28 AM2/5/09
to Twitter Development Talk
Thanks Alex,

I too, would like to see this return userids AND usernames.

-Joel

On Feb 3, 5:01 pm, Alex Payne <a...@twitter.com> wrote:
> Happy to announce two new API methods today, delivered in response to
> developer demand for an easier way to keep tabs on users' social graphs.
> The methods, /friends/ids and /followers/ids, return the entire list of
> numeric user IDs for a user's set of followed and following users,
> respectively. Responses to these methods are cached until the user's
> social graph changes. The responses come direct from our denormalized
> list data stores, and should be reasonably fast even for users with a
> large number of followers/follows.
>
> These new methods are most useful for services that are maintaining a
> cache of user details. If you see a user ID that you don't have cached,
> you'll have to call /users/show to retrieve that user's details. But for
> services with large user bases, or those that simply want to diff a
> user's social graph over time, we hope these methods will come in handy.
>
> You can find the documentation athttp://apiwiki.twitter.com/REST-API-Documentation#SocialGraphMethods.

dougw

unread,
Feb 5, 2009, 11:51:57 AM2/5/09
to Twitter Development Talk
For all those wanting id AND username attributes to be returned with
these new methods, be sure to head over to
http://code.google.com/p/twitter-api/issues/detail?id=265 and vote
(click the star) to signal your support.

@dougw

Alex Payne

unread,
Feb 5, 2009, 1:19:42 PM2/5/09
to twitter-deve...@googlegroups.com
The reason why we can provide the list of IDs without any pagination is
that it comes directly from our denormalized list data store, and
requires no joining, either in SQL or at the application layer. As soon
as we pull in data like screen_name that's sitting elsewhere in our
architecture, the response time slows down drastically.

So while I do understand that it'd be more convenient, my hunch is that
the quality of service for such a method would be intolerable.

Stuart

unread,
Feb 5, 2009, 1:23:03 PM2/5/09
to twitter-deve...@googlegroups.com
Perhaps a better solution might be the ability to pass multiple IDs to
the user/show method so you could retrieve the details of more than
one user in one request, maybe with a limit to the maximum you can get
in one go.

-Stuart

2009/2/5 Alex Payne <al...@twitter.com>:

--
http://stut.net/

Ninjamonk

unread,
Feb 5, 2009, 5:47:20 PM2/5/09
to Twitter Development Talk
Hey Alex, any chance of adding a way of looking up the user name to
the search api instead then?

On Feb 5, 6:19 pm, Alex Payne <a...@twitter.com> wrote:
> The reason why we can provide the list of IDs without any pagination is
> that it comes directly from our denormalized list data store, and
> requires no joining, either in SQL or at the application layer. As soon
> as we pull in data like screen_name that's sitting elsewhere in our
> architecture, the response time slows down drastically.
>
> So while I do understand that it'd be more convenient, my hunch is that
> the quality of service for such a method would be intolerable.
>
>
>
> dougw wrote:
> > For all those wanting id AND username attributes to be returned with
> > these new methods, be sure to head over to
> >http://code.google.com/p/twitter-api/issues/detail?id=265and vote

Alex Payne

unread,
Feb 5, 2009, 5:49:52 PM2/5/09
to twitter-deve...@googlegroups.com
You can presently look up usernames in the Search API. What in
particular are you looking for?

Ninjamonk

unread,
Feb 6, 2009, 6:16:17 AM2/6/09
to Twitter Development Talk
sorry badly put, I meant via user id, search via user id so like FROM:
342342342 etc returns the same as say FROM: ninjamonk.

dougw

unread,
Feb 6, 2009, 9:38:33 AM2/6/09
to Twitter Development Talk
Ninjamonk, I second what you are asking for here: a search API
addition to allow ids as valid queries.

In my code, I represent users by their static ids rather than their
screen_names but when doing a search I have to convert back. It would
be nice to have the flexability to do search through both user id and
user screen name.

Does anyone else see a need here?

@dougw

nicdev

unread,
Feb 6, 2009, 10:56:56 AM2/6/09
to Twitter Development Talk
I haven't tried it out yet, but it's exactly what I needed. Thanks!

Sylvain Munaut

unread,
Feb 6, 2009, 1:23:53 PM2/6/09
to Twitter Development Talk
On Feb 6, 3:38 pm, dougw <igu...@gmail.com> wrote:
> Ninjamonk, I second what you are asking for here: a search API
> addition to allow ids as valid queries.

It would be a good start if the user_id used in the search API were
the same as the user_id in the rest of the API :)
Scheduled for V2 apparently ...

Sylvain

Stuart

unread,
Feb 7, 2009, 2:53:45 PM2/7/09
to twitter-deve...@googlegroups.com, twitter-ap...@googlegroups.com
Loving these new methods, but was disappointed to see that they don't
support the If-Modified-Since header. Any chance of that to reduce
data transfer at both ends and unnecessary processing at our end?

-Stuart

2009/2/4 Alex Payne <al...@twitter.com>:


>
> Happy to announce two new API methods today, delivered in response to
> developer demand for an easier way to keep tabs on users' social graphs. The
> methods, /friends/ids and /followers/ids, return the entire list of numeric
> user IDs for a user's set of followed and following users, respectively.
> Responses to these methods are cached until the user's social graph changes.
> The responses come direct from our denormalized list data stores, and should
> be reasonably fast even for users with a large number of followers/follows.
>
> These new methods are most useful for services that are maintaining a cache
> of user details. If you see a user ID that you don't have cached, you'll
> have to call /users/show to retrieve that user's details. But for services
> with large user bases, or those that simply want to diff a user's social
> graph over time, we hope these methods will come in handy.
>
> You can find the documentation at
> http://apiwiki.twitter.com/REST-API-Documentation#SocialGraphMethods.
>
> --

Alex Payne

unread,
Feb 7, 2009, 3:50:40 PM2/7/09
to twitter-deve...@googlegroups.com, twitter-ap...@googlegroups.com
If you file a ticket, I can check with our engineers to see how performance would look if we try to support that filter on the collection we return.

Stuart

unread,
Feb 7, 2009, 4:11:15 PM2/7/09
to twitter-deve...@googlegroups.com, twitter-ap...@googlegroups.com

Shannon Whitley

unread,
Feb 11, 2009, 8:08:19 PM2/11/09
to Twitter Development Talk
I was really looking forward to implementing these API methods. After
dealing with the pagination for so long, this seemed perfect. Then I
realized that I have to call "/users/show" for every user to get the
most current bio etc. So instead of making 10 calls for a user with
1,000 followers, I'm now making 1,000 calls for that same user. It
won't take long to hit the 20K limit. I suppose I could stagger the
user updates, but I really need the current data.

Am I coming at this from the right angle? Does anyone see any flaws
in my thinking here?



On Feb 3, 5:01 pm, Alex Payne <a...@twitter.com> wrote:
> Happy to announce two new API methods today, delivered in response to
> developer demand for an easier way to keep tabs on users' social graphs.
> The methods, /friends/idsand /followers/ids, return the entire list of
> numeric userIDsfor a user's set of followed and following users,
> respectively. Responses to these methods are cached until the user's
> social graph changes. The responses come direct from our denormalized
> list data stores, and should be reasonably fast even for users with a
> large number of followers/follows.
>
> These new methods are most useful for services that are maintaining a
> cache of user details. If you see a user ID that you don't have cached,
> you'll have to call /users/show to retrieve that user's details. But for
> services with large user bases, or those that simply want to diff a
> user's social graph over time, we hope these methods will come in handy.
>
> You can find the documentation athttp://apiwiki.twitter.com/REST-API-Documentation#SocialGraphMethods.

pnoeric

unread,
Feb 22, 2009, 2:19:12 AM2/22/09
to Twitter Development Talk
Just wanted to mention-- updated my app to use these new methods, and
now one routine that took OVER 14 minutes to run is now under 1
minute... what an excellent difference. Thanks y'all. -Eric


On Feb 3, 5:01 pm, Alex Payne <a...@twitter.com> wrote:
> Happy to announce two new API methods today, delivered in response to
> developer demand for an easier way to keep tabs on users'socialgraphs.
> The methods, /friends/ids and /followers/ids, return the entire list of
> numeric user IDs for a user's set of followed and following users,
> respectively. Responses to these methods are cached until the user'ssocialgraphchanges. The responses come direct from our denormalized
> list data stores, and should be reasonably fast even for users with a
> large number of followers/follows.
>
> These new methods are most useful for services that are maintaining a
> cache of user details. If you see a user ID that you don't have cached,
> you'll have to call /users/show to retrieve that user's details. But for
> services with large user bases, or those that simply want to diff a
> user'ssocialgraphover time, we hope these methods will come in handy.
Reply all
Reply to author
Forward
0 new messages