The Particle Lab

Building WordStream

By Marc, 26 Jan 2015

Last week I spent a couple of hours playing with some new technology and built wordstream, a real-time word cloud generated from the twitter sample stream. Here’s how.

The twitter streaming APIs are a very efficient way of having the tweets you’re interested in pushed to you. For example you can use the filter endpoint to have tweets matching your filter (author, hashtag, keywords etc) but for this I was more interested in the sample endpoint which sends out about 1% of all public tweets. This endpoint does however have some limitations:

Here’s a quick example (capturing the stream for about 5 seconds resulting in 1.3 MB of data, I’ve shown just the first ~1000 lines here, a sample of the sample you could say) of the streaming API data:

{ created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954651971600,
    id_str: '559747954651971584',
    text: 'Мосгорсуд оставил под арестом до 16 апреля Александра Кольченко, фигуранта дела ...',
    source: '<a href="http://ifttt.com" rel="nofollow">IFTTT</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 2687442584,
       id_str: '2687442584',
       name: 'Галина Никандровa',
       screen_name: 'Byce6A',
       location: '',
       url: null,
       description: null,
       protected: false,
       verified: false,
       followers_count: 210,
       friends_count: 121,
       listed_count: 1,
       favourites_count: 0,
       statuses_count: 73725,
       created_at: 'Mon Jul 28 12:45:30 +0000 2014',
       utc_offset: null,
       time_zone: null,
       geo_enabled: false,
       lang: 'ru',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: 'C0DEED',
       profile_background_image_url: 'http://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_image_url_https: 'https://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_tile: false,
       profile_link_color: '0084B4',
       profile_sidebar_border_color: 'C0DEED',
       profile_sidebar_fill_color: 'DDEEF6',
       profile_text_color: '333333',
       profile_use_background_image: true,
       profile_image_url: 'http://abs.twimg.com/sticky/default_profile_images/default_profile_1_normal.png',
       profile_image_url_https: 'https://abs.twimg.com/sticky/default_profile_images/default_profile_1_normal.png',
       default_profile: true,
       default_profile_image: true,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [],
       trends: [],
       urls: [],
       user_mentions: [],
       symbols: [] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'ru',
    timestamp_ms: '1422289286660' }
  { created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954639384600,
    id_str: '559747954639384577',
    text: 'Beautiful life is so much better than Carry you tbh',
    source: '<a href="http://twitter.com" rel="nofollow">Twitter Web Client</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 2974152997,
       id_str: '2974152997',
       name: 'Sandra Young',
       screen_name: 'edwardalazobuy1',
       location: 'West Virginia',
       url: 'http://optimizedirectory.com/',
       description: '1D / Glee / T-Swizzle / Narnia / Criminal Minds / KSS 8 / Lucky #18/ #23 / #24 / Directioner / MATTHEW GRAY GUBLER FOR DA WIN! / Louis\' pants',
       protected: false,
       verified: false,
       followers_count: 0,
       friends_count: 1,
       listed_count: 0,
       favourites_count: 0,
       statuses_count: 37,
       created_at: 'Sun Jan 11 06:10:53 +0000 2015',
       utc_offset: null,
       time_zone: null,
       geo_enabled: false,
       lang: 'en',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: 'C0DEED',
       profile_background_image_url: 'http://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_image_url_https: 'https://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_tile: false,
       profile_link_color: '0084B4',
       profile_sidebar_border_color: 'C0DEED',
       profile_sidebar_fill_color: 'DDEEF6',
       profile_text_color: '333333',
       profile_use_background_image: true,
       profile_image_url: 'http://pbs.twimg.com/profile_images/559450280236830720/fGI9TXLt_normal.png',
       profile_image_url_https: 'https://pbs.twimg.com/profile_images/559450280236830720/fGI9TXLt_normal.png',
       profile_banner_url: 'https://pbs.twimg.com/profile_banners/2974152997/1422261339',
       default_profile: true,
       default_profile_image: false,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [],
       trends: [],
       urls: [],
       user_mentions: [],
       symbols: [] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'en',
    timestamp_ms: '1422289286657' }
  { created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954672943100,
    id_str: '559747954672943104',
    text: 'Saints win 2-0! Enppi are 0-0 so double chance looking good on this one too.',
    source: '<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 2960224947,
       id_str: '2960224947',
       name: 'The Secret Tipster',
       screen_name: 'Secret_Tipster_',
       location: '',
       url: null,
       description: 'FREE betting tips and £10-£1,000 challenges! \n\n5pts - Strong tip (high stakes)\n3pts - Good tip (medium stakes)\n1pt   - Fair tip (low stakes)',
       protected: false,
       verified: false,
       followers_count: 343,
       friends_count: 1588,
       listed_count: 2,
       favourites_count: 104,
       statuses_count: 290,
       created_at: 'Sun Jan 04 14:09:31 +0000 2015',
       utc_offset: 0,
       time_zone: 'London',
       geo_enabled: false,
       lang: 'en-gb',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: '000000',
       profile_background_image_url: 'http://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_image_url_https: 'https://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_tile: false,
       profile_link_color: '89C9FA',
       profile_sidebar_border_color: '000000',
       profile_sidebar_fill_color: '000000',
       profile_text_color: '000000',
       profile_use_background_image: false,
       profile_image_url: 'http://pbs.twimg.com/profile_images/551742687452229634/Q2rfimMq_normal.png',
       profile_image_url_https: 'https://pbs.twimg.com/profile_images/551742687452229634/Q2rfimMq_normal.png',
       default_profile: false,
       default_profile_image: false,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [],
       trends: [],
       urls: [],
       user_mentions: [],
       symbols: [] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'en',
    timestamp_ms: '1422289286665' }
  { created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954647793660,
    id_str: '559747954647793666',
    text: '3 hours of oral anatomy on a Monday morning... Bad idea #confusedfordays',
    source: '<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 481348971,
       id_str: '481348971',
       name: 'Sara Gallagher',
       screen_name: 'Saraaa08',
       location: '',
       url: null,
       description: 'Life is about not knowing, having to change, taking the moment and making the best of it, without knowing what\'s going to happen next.',
       protected: false,
       verified: false,
       followers_count: 86,
       friends_count: 86,
       listed_count: 0,
       favourites_count: 763,
       statuses_count: 2007,
       created_at: 'Thu Feb 02 16:48:17 +0000 2012',
       utc_offset: -21600,
       time_zone: 'Central Time (US & Canada)',
       geo_enabled: false,
       lang: 'en',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: 'E113F0',
       profile_background_image_url: 'http://pbs.twimg.com/profile_background_images/633347123/p8t7222tk4jxb2l437zv.jpeg',
       profile_background_image_url_https: 'https://pbs.twimg.com/profile_background_images/633347123/p8t7222tk4jxb2l437zv.jpeg',
       profile_background_tile: true,
       profile_link_color: '9F16CC',
       profile_sidebar_border_color: '65B0DA',
       profile_sidebar_fill_color: '7AC3EE',
       profile_text_color: '3D1957',
       profile_use_background_image: true,
       profile_image_url: 'http://pbs.twimg.com/profile_images/445991802739249152/8LSUaeA8_normal.jpeg',
       profile_image_url_https: 'https://pbs.twimg.com/profile_images/445991802739249152/8LSUaeA8_normal.jpeg',
       profile_banner_url: 'https://pbs.twimg.com/profile_banners/481348971/1376020262',
       default_profile: false,
       default_profile_image: false,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [ [Object] ],
       trends: [],
       urls: [],
       user_mentions: [],
       symbols: [] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'en',
    timestamp_ms: '1422289286659' }
  { created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954660376600,
    id_str: '559747954660376576',
    text: 'Don\'t twerk if you don\'t have a butt. It is that simple',
    source: '<a href="http://twitter.com" rel="nofollow">Twitter Web Client</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 2982012311,
       id_str: '2982012311',
       name: 'Ronald Garcia',
       screen_name: 'christo41542623',
       location: 'in the airport Agh',
       url: 'http://hardcorewebdir.info/',
       description: 'What we do here echoes in eternity.',
       protected: false,
       verified: false,
       followers_count: 0,
       friends_count: 1,
       listed_count: 0,
       favourites_count: 0,
       statuses_count: 25,
       created_at: 'Sat Jan 17 01:42:36 +0000 2015',
       utc_offset: null,
       time_zone: null,
       geo_enabled: false,
       lang: 'en',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: 'C0DEED',
       profile_background_image_url: 'http://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_image_url_https: 'https://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_tile: false,
       profile_link_color: '0084B4',
       profile_sidebar_border_color: 'C0DEED',
       profile_sidebar_fill_color: 'DDEEF6',
       profile_text_color: '333333',
       profile_use_background_image: true,
       profile_image_url: 'http://pbs.twimg.com/profile_images/559440775339966464/L9Jleb81_normal.png',
       profile_image_url_https: 'https://pbs.twimg.com/profile_images/559440775339966464/L9Jleb81_normal.png',
       profile_banner_url: 'https://pbs.twimg.com/profile_banners/2982012311/1422255160',
       default_profile: true,
       default_profile_image: false,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [],
       trends: [],
       urls: [],
       user_mentions: [],
       symbols: [] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'en',
    timestamp_ms: '1422289286662' }
  { created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954668757000,
    id_str: '559747954668756994',
    text: 'RT @Viccent22: Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar. Dan, lagi-lagi harus …',
    source: '<a href="http://www.writelonger.com" rel="nofollow">Write Longer</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 1851111336,
       id_str: '1851111336',
       name: 'Ann!',
       screen_name: 'PipitAnnisaF',
       location: 'Purbalingga-Gombong-Bandung',
       url: null,
       description: 'Bismillahirohmanirrohim... Allahumma yassir wa laa tu\'assir... Ya Allah,permudahkanlah jangan dipersulit. {Ab Ann}',
       protected: false,
       verified: false,
       followers_count: 400,
       friends_count: 196,
       listed_count: 1,
       favourites_count: 70,
       statuses_count: 13273,
       created_at: 'Tue Sep 10 10:57:46 +0000 2013',
       utc_offset: 25200,
       time_zone: 'Bangkok',
       geo_enabled: true,
       lang: 'id',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: 'FCEBB6',
       profile_background_image_url: 'http://pbs.twimg.com/profile_background_images/378800000115095569/094796ad031c0de4fdb006f3de3c197a.png',
       profile_background_image_url_https: 'https://pbs.twimg.com/profile_background_images/378800000115095569/094796ad031c0de4fdb006f3de3c197a.png',
       profile_background_tile: true,
       profile_link_color: 'CE7834',
       profile_sidebar_border_color: '000000',
       profile_sidebar_fill_color: '78C0A8',
       profile_text_color: '5E412F',
       profile_use_background_image: true,
       profile_image_url: 'http://pbs.twimg.com/profile_images/556377564667604993/uFr5rOI__normal.jpeg',
       profile_image_url_https: 'https://pbs.twimg.com/profile_images/556377564667604993/uFr5rOI__normal.jpeg',
       profile_banner_url: 'https://pbs.twimg.com/profile_banners/1851111336/1413637653',
       default_profile: false,
       default_profile_image: false,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweeted_status:
     { created_at: 'Mon Jan 26 16:20:24 +0000 2015',
       id: 559747692579262460,
       id_str: '559747692579262465',
       text: 'Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar Sabar. Dan, lagi-lagi harus sabar :\')',
       source: '<a href="http://www.hootsuite.com" rel="nofollow">Hootsuite</a>',
       truncated: false,
       in_reply_to_status_id: null,
       in_reply_to_status_id_str: null,
       in_reply_to_user_id: null,
       in_reply_to_user_id_str: null,
       in_reply_to_screen_name: null,
       user:
        { id: 812104176,
          id_str: '812104176',
          name: 'Hantu TimeLine',
          screen_name: 'Viccent22',
          location: 'FOLLOW MY INSTAGRAM: Viccent22',
          url: 'http://ask.fm/viccenttt',
          description: 'Jangan follow, nanti naksir(•̯͡.•̯͡) | Contact Us: ☎087770123452 / contactme22222@gmail.com',
          protected: false,
          verified: false,
          followers_count: 850235,
          friends_count: 0,
          listed_count: 355,
          favourites_count: 698,
          statuses_count: 10579,
          created_at: 'Sun Sep 09 02:02:12 +0000 2012',
          utc_offset: 25200,
          time_zone: 'Jakarta',
          geo_enabled: false,
          lang: 'en',
          contributors_enabled: false,
          is_translator: false,
          profile_background_color: '080808',
          profile_background_image_url: 'http://pbs.twimg.com/profile_background_images/747931822/ebd2c3137e9dcbd9361b1b163fd21481.png',
          profile_background_image_url_https: 'https://pbs.twimg.com/profile_background_images/747931822/ebd2c3137e9dcbd9361b1b163fd21481.png',
          profile_background_tile: false,
          profile_link_color: 'F00E0E',
          profile_sidebar_border_color: '000000',
          profile_sidebar_fill_color: 'F6F6F6',
          profile_text_color: '333333',
          profile_use_background_image: true,
          profile_image_url: 'http://pbs.twimg.com/profile_images/477404179636289537/gNRRvecD_normal.jpeg',
          profile_image_url_https: 'https://pbs.twimg.com/profile_images/477404179636289537/gNRRvecD_normal.jpeg',
          profile_banner_url: 'https://pbs.twimg.com/profile_banners/812104176/1402835276',
          default_profile: false,
          default_profile_image: false,
          following: null,
          follow_request_sent: null,
          notifications: null },
       geo: null,
       coordinates: null,
       place: null,
       contributors: null,
       retweet_count: 33,
       favorite_count: 0,
       entities:
        { hashtags: [],
          trends: [],
          urls: [],
          user_mentions: [],
          symbols: [] },
       favorited: false,
       retweeted: false,
       possibly_sensitive: true,
       filter_level: 'low',
       lang: 'in' },
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [],
       trends: [],
       urls: [],
       user_mentions: [ [Object] ],
       symbols: [] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'in',
    timestamp_ms: '1422289286664' }
  { created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954639372300,
    id_str: '559747954639372290',
    text: '🍀@NiallOfficial hi sunshine!🍀\nHow are u? I hope u\'re fine😉\n🌸I love u so much, can u follow me?🌸\nU\'re my everything.\nu saved me. 💕😭 x 3.384',
    source: '<a href="http://twitter.com" rel="nofollow">Twitter Web Client</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 1033575770,
       id_str: '1033575770',
       name: 'nialler ',
       screen_name: 'nixalley',
       location: '020714',
       url: null,
       description: 'if u loved me why\'d u leave me? || i\'m in love with p and n',
       protected: false,
       verified: false,
       followers_count: 3039,
       friends_count: 2385,
       listed_count: 12,
       favourites_count: 2460,
       statuses_count: 52990,
       created_at: 'Mon Dec 24 22:29:10 +0000 2012',
       utc_offset: 3600,
       time_zone: 'Warsaw',
       geo_enabled: true,
       lang: 'en-gb',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: 'FFFFFF',
       profile_background_image_url: 'http://pbs.twimg.com/profile_background_images/452216615522676736/vkqDYOOn.png',
       profile_background_image_url_https: 'https://pbs.twimg.com/profile_background_images/452216615522676736/vkqDYOOn.png',
       profile_background_tile: false,
       profile_link_color: '9266CC',
       profile_sidebar_border_color: 'FFFFFF',
       profile_sidebar_fill_color: 'EFEFEF',
       profile_text_color: '333333',
       profile_use_background_image: true,
       profile_image_url: 'http://pbs.twimg.com/profile_images/551491161215361025/kVwuzBr6_normal.png',
       profile_image_url_https: 'https://pbs.twimg.com/profile_images/551491161215361025/kVwuzBr6_normal.png',
       profile_banner_url: 'https://pbs.twimg.com/profile_banners/1033575770/1420316329',
       default_profile: false,
       default_profile_image: false,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [],
       trends: [],
       urls: [],
       user_mentions: [ [Object] ],
       symbols: [] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'en',
    timestamp_ms: '1422289286657' }
  { created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954651959300,
    id_str: '559747954651959296',
    text: 'أستغفر الله العظيم الذي لا إله إلا هو الحي القيوم و أتوب إليه http://t.co/UwtV9SLroN',
    source: '<a href="http://knzmuslim.com" rel="nofollow">knzmuslim كنز المسلم</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 2807986238,
       id_str: '2807986238',
       name: 'Salman',
       screen_name: 'alghamdisalman1',
       location: '',
       url: null,
       description: 'كنت ولا زلت وسأظل اعشق الكيان الازرق معاك على الحلوه والمره يا زعيم',
       protected: false,
       verified: false,
       followers_count: 325,
       friends_count: 417,
       listed_count: 0,
       favourites_count: 4,
       statuses_count: 2605,
       created_at: 'Sat Sep 13 18:59:35 +0000 2014',
       utc_offset: null,
       time_zone: null,
       geo_enabled: false,
       lang: 'ar',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: 'C0DEED',
       profile_background_image_url: 'http://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_image_url_https: 'https://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_tile: false,
       profile_link_color: '0084B4',
       profile_sidebar_border_color: 'C0DEED',
       profile_sidebar_fill_color: 'DDEEF6',
       profile_text_color: '333333',
       profile_use_background_image: true,
       profile_image_url: 'http://pbs.twimg.com/profile_images/540117613884366848/hsp1dn2e_normal.jpeg',
       profile_image_url_https: 'https://pbs.twimg.com/profile_images/540117613884366848/hsp1dn2e_normal.jpeg',
       profile_banner_url: 'https://pbs.twimg.com/profile_banners/2807986238/1414873049',
       default_profile: true,
       default_profile_image: false,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [],
       trends: [],
       urls: [ [Object] ],
       user_mentions: [],
       symbols: [] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'ar',
    timestamp_ms: '1422289286660' }
  { created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954651971600,
    id_str: '559747954651971585',
    text: '(ما قدروا الله حق قدره إن الله لقوي عزيز) [الحج:74] http://t.co/muW1xfBOGo',
    source: '<a href="http://qurani.tv" rel="nofollow">تطبيق قرآني</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 2190909662,
       id_str: '2190909662',
       name: 'لعله يشفع لي⛅️',
       screen_name: 'for_mona3',
       location: '@xMonaAlx ☀️',
       url: 'http://www.tvquran.com',
       description: 'اذكروه يذكركم',
       protected: false,
       verified: false,
       followers_count: 5,
       friends_count: 2,
       listed_count: 0,
       favourites_count: 1,
       statuses_count: 9791,
       created_at: 'Tue Nov 12 19:25:01 +0000 2013',
       utc_offset: null,
       time_zone: null,
       geo_enabled: false,
       lang: 'ar',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: 'C0DEED',
       profile_background_image_url: 'http://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_image_url_https: 'https://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_tile: false,
       profile_link_color: '0084B4',
       profile_sidebar_border_color: 'C0DEED',
       profile_sidebar_fill_color: 'DDEEF6',
       profile_text_color: '333333',
       profile_use_background_image: true,
       profile_image_url: 'http://pbs.twimg.com/profile_images/416745923730235392/YDAesMzC_normal.jpeg',
       profile_image_url_https: 'https://pbs.twimg.com/profile_images/416745923730235392/YDAesMzC_normal.jpeg',
       profile_banner_url: 'https://pbs.twimg.com/profile_banners/2190909662/1384714039',
       default_profile: true,
       default_profile_image: false,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [],
       trends: [],
       urls: [ [Object] ],
       user_mentions: [],
       symbols: [] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'ar',
    timestamp_ms: '1422289286660' }
  { created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954651963400,
    id_str: '559747954651963395',
    text: 'RT @ilknurBasli: 18509ihtiyacla Lideringilizce 51',
    source: '<a href="http://twitter.com/download/android" rel="nofollow">Twitter for Android</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 2906151585,
       id_str: '2906151585',
       name: 'Begum Ozsahin',
       screen_name: 'bbggmmssrrkknn',
       location: '',
       url: null,
       description: null,
       protected: false,
       verified: false,
       followers_count: 36,
       friends_count: 12,
       listed_count: 0,
       favourites_count: 5,
       statuses_count: 1896,
       created_at: 'Fri Dec 05 09:57:56 +0000 2014',
       utc_offset: null,
       time_zone: null,
       geo_enabled: false,
       lang: 'tr',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: 'C0DEED',
       profile_background_image_url: 'http://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_image_url_https: 'https://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_tile: false,
       profile_link_color: '0084B4',
       profile_sidebar_border_color: 'C0DEED',
       profile_sidebar_fill_color: 'DDEEF6',
       profile_text_color: '333333',
       profile_use_background_image: true,
       profile_image_url: 'http://pbs.twimg.com/profile_images/557186002578866176/dGheKKpE_normal.jpeg',
       profile_image_url_https: 'https://pbs.twimg.com/profile_images/557186002578866176/dGheKKpE_normal.jpeg',
       default_profile: true,
       default_profile_image: false,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweeted_status:
     { created_at: 'Mon Jan 26 15:54:40 +0000 2015',
       id: 559741218863992800,
       id_str: '559741218863992833',
       text: '18509ihtiyacla Lideringilizce 51',
       source: '<a href="http://twitter.com" rel="nofollow">Twitter Web Client</a>',
       truncated: false,
       in_reply_to_status_id: null,
       in_reply_to_status_id_str: null,
       in_reply_to_user_id: null,
       in_reply_to_user_id_str: null,
       in_reply_to_screen_name: null,
       user:
        { id: 1599517687,
          id_str: '1599517687',
          name: 'ilknur BAŞLI',
          screen_name: 'ilknurBasli',
          location: '',
          url: null,
          description: null,
          protected: false,
          verified: false,
          followers_count: 778,
          friends_count: 852,
          listed_count: 0,
          favourites_count: 712,
          statuses_count: 11538,
          created_at: 'Tue Jul 16 23:24:47 +0000 2013',
          utc_offset: null,
          time_zone: null,
          geo_enabled: false,
          lang: 'tr',
          contributors_enabled: false,
          is_translator: false,
          profile_background_color: 'C0DEED',
          profile_background_image_url: 'http://abs.twimg.com/images/themes/theme1/bg.png',
          profile_background_image_url_https: 'https://abs.twimg.com/images/themes/theme1/bg.png',
          profile_background_tile: false,
          profile_link_color: '0084B4',
          profile_sidebar_border_color: 'C0DEED',
          profile_sidebar_fill_color: 'DDEEF6',
          profile_text_color: '333333',
          profile_use_background_image: true,
          profile_image_url: 'http://pbs.twimg.com/profile_images/511566148530618368/V4WS62lr_normal.jpeg',
          profile_image_url_https: 'https://pbs.twimg.com/profile_images/511566148530618368/V4WS62lr_normal.jpeg',
          profile_banner_url: 'https://pbs.twimg.com/profile_banners/1599517687/1418137237',
          default_profile: true,
          default_profile_image: false,
          following: null,
          follow_request_sent: null,
          notifications: null },
       geo: null,
       coordinates: null,
       place: null,
       contributors: null,
       retweet_count: 6,
       favorite_count: 1,
       entities:
        { hashtags: [],
          trends: [],
          urls: [],
          user_mentions: [],
          symbols: [] },
       favorited: false,
       retweeted: false,
       possibly_sensitive: false,
       filter_level: 'low',
       lang: 'tr' },
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [],
       trends: [],
       urls: [],
       user_mentions: [ [Object] ],
       symbols: [] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'tr',
    timestamp_ms: '1422289286660' }
  { created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954660368400,
    id_str: '559747954660368385',
    text: 'You\'re not clever you fucking Greb',
    source: '<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 2427830686,
       id_str: '2427830686',
       name: '⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀',
       screen_name: 'RAFAELUDX',
       location: '',
       url: 'http://www.unorthodoxrepublic.bigcartel.com',
       description: 'IG:RafaelUDX\nSC:Rafaellol',
       protected: false,
       verified: false,
       followers_count: 1124,
       friends_count: 379,
       listed_count: 1,
       favourites_count: 895,
       statuses_count: 2116,
       created_at: 'Fri Mar 21 22:52:37 +0000 2014',
       utc_offset: 3600,
       time_zone: 'Amsterdam',
       geo_enabled: false,
       lang: 'en-gb',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: '000000',
       profile_background_image_url: 'http://pbs.twimg.com/profile_background_images/468511827144364033/CKxYF1fx.jpeg',
       profile_background_image_url_https: 'https://pbs.twimg.com/profile_background_images/468511827144364033/CKxYF1fx.jpeg',
       profile_background_tile: false,
       profile_link_color: 'FFFFFF',
       profile_sidebar_border_color: '000000',
       profile_sidebar_fill_color: 'DDEEF6',
       profile_text_color: '333333',
       profile_use_background_image: true,
       profile_image_url: 'http://pbs.twimg.com/profile_images/558565834415931392/OhRnr15H_normal.jpeg',
       profile_image_url_https: 'https://pbs.twimg.com/profile_images/558565834415931392/OhRnr15H_normal.jpeg',
       profile_banner_url: 'https://pbs.twimg.com/profile_banners/2427830686/1422007563',
       default_profile: false,
       default_profile_image: false,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [],
       trends: [],
       urls: [],
       user_mentions: [],
       symbols: [] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'en',
    timestamp_ms: '1422289286662' }
  { created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954651959300,
    id_str: '559747954651959297',
    text: 'RT @JeuneAthena: Hommage à Zac Evans, anglais de 19 ans, tué à coup de machette pour protéger des filles.\nLe multiculturalisme tue.\n\n- http…',
    source: '<a href="http://twitter.com/download/android" rel="nofollow">Twitter for Android</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 2190207191,
       id_str: '2190207191',
       name: 'artigouha herail ن ',
       screen_name: 'HerailArtigouha',
       location: 'languedoc/midi-Pyrénées',
       url: null,
       description: 'après mes enfants j\'ai élevé mon vin! maintenant:3ème vie à inventer...',
       protected: false,
       verified: false,
       followers_count: 1139,
       friends_count: 1161,
       listed_count: 26,
       favourites_count: 34075,
       statuses_count: 100572,
       created_at: 'Thu Nov 21 17:52:30 +0000 2013',
       utc_offset: null,
       time_zone: null,
       geo_enabled: true,
       lang: 'fr',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: 'C0DEED',
       profile_background_image_url: 'http://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_image_url_https: 'https://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_tile: false,
       profile_link_color: '0084B4',
       profile_sidebar_border_color: 'C0DEED',
       profile_sidebar_fill_color: 'DDEEF6',
       profile_text_color: '333333',
       profile_use_background_image: true,
       profile_image_url: 'http://pbs.twimg.com/profile_images/547124242555609088/6SJm4nu4_normal.jpeg',
       profile_image_url_https: 'https://pbs.twimg.com/profile_images/547124242555609088/6SJm4nu4_normal.jpeg',
       profile_banner_url: 'https://pbs.twimg.com/profile_banners/2190207191/1419279829',
       default_profile: true,
       default_profile_image: false,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweeted_status:
     { created_at: 'Mon Jan 26 13:43:53 +0000 2015',
       id: 559708303853641700,
       id_str: '559708303853641728',
       text: 'Hommage à Zac Evans, anglais de 19 ans, tué à coup de machette pour protéger des filles.\nLe multiculturalisme tue.\n\n- http://t.co/r06ncd1meg',
       source: '<a href="http://twitter.com/download/android" rel="nofollow">Twitter for Android</a>',
       truncated: false,
       in_reply_to_status_id: null,
       in_reply_to_status_id_str: null,
       in_reply_to_user_id: null,
       in_reply_to_user_id_str: null,
       in_reply_to_screen_name: null,
       user:
        { id: 576201752,
          id_str: '576201752',
          name: 'Jeune Athéna ',
          screen_name: 'JeuneAthena',
          location: 'ELSASS, Europa',
          url: null,
          description: 'Européenne de souche de 21 ans | Nationaliste identitaire | Hostile aux #GrandRemplacement, multiculturalisme et égalitarisme. PEGIDA!',
          protected: false,
          verified: false,
          followers_count: 8868,
          friends_count: 567,
          listed_count: 128,
          favourites_count: 7788,
          statuses_count: 19658,
          created_at: 'Thu May 10 12:44:30 +0000 2012',
          utc_offset: 7200,
          time_zone: 'Athens',
          geo_enabled: true,
          lang: 'en',
          contributors_enabled: false,
          is_translator: false,
          profile_background_color: 'B06121',
          profile_background_image_url: 'http://pbs.twimg.com/profile_background_images/495015736893403136/nVOACylL.jpeg',
          profile_background_image_url_https: 'https://pbs.twimg.com/profile_background_images/495015736893403136/nVOACylL.jpeg',
          profile_background_tile: false,
          profile_link_color: 'D5842B',
          profile_sidebar_border_color: '000000',
          profile_sidebar_fill_color: 'DDEEF6',
          profile_text_color: '333333',
          profile_use_background_image: false,
          profile_image_url: 'http://pbs.twimg.com/profile_images/542640579713458176/32Zao_II_normal.jpeg',
          profile_image_url_https: 'https://pbs.twimg.com/profile_images/542640579713458176/32Zao_II_normal.jpeg',
          profile_banner_url: 'https://pbs.twimg.com/profile_banners/576201752/1415484544',
          default_profile: false,
          default_profile_image: false,
          following: null,
          follow_request_sent: null,
          notifications: null },
       geo: null,
       coordinates: null,
       place: null,
       contributors: null,
       retweet_count: 70,
       favorite_count: 22,
       entities:
        { hashtags: [],
          trends: [],
          urls: [],
          user_mentions: [],
          symbols: [],
          media: [Object] },
       extended_entities: { media: [Object] },
       favorited: false,
       retweeted: false,
       possibly_sensitive: false,
       filter_level: 'low',
       lang: 'fr' },
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [],
       trends: [],
       urls: [],
       user_mentions: [ [Object] ],
       symbols: [],
       media: [ [Object] ] },
    extended_entities: { media: [ [Object] ] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'fr',
    timestamp_ms: '1422289286660' }

Here’s a few things to note:

All of this meant that it made sense to build a simple back-end service/proxy that created a single streaming connection, processed this data and fed a far more condensed amount of data out to the browser(s). I chose to build something with node.js.

First we need to get the data out of the streaming API. I found a npm module called node-tweet-stream that worked with the filter endpoint, and with a little butchery was able to hook it up to the sample API instead.

var twitter = require('./twitter-stream'), //
    stream;

stream = new twitter({
  consumer_key: 'xxx',
  consumer_secret: 'xxx',
  token: 'xxx',
  token_secret: 'xxx'
});

stream.on('tweet', function(tweet) {
  console.log(tweet);
});

stream.connect();

I often use Heroku for hosting small things like this and Heroku encourages you to store as much of the application configuration as possible in the environment rather your application code respository. To manage this in my Ruby projects I use dotenv to allow me to keep such configuration in a .env file locally (excluding this from the source control). I was very pleased to find such functionality also exists for developing in node. A quick install of the dotenv npm module and a simple require and it was working here.

Logging things out to the console is great for debugging things but no real use. To get the data out to a browser I started to build a simple express app as I’d had some experience with this before but something reminded me of web sockets and socket.io so I thought I’d try playing with them. Again, all that was required was another install/require and a couple of extra lines and now we have tweets being proxied through to the browser(s). The code was now looking like this:

var app = require('express')(),
    dotenv = require('dotenv'),
    server = require('http').Server(app),
    io = require('socket.io')(server),
    twitter = require('./twitter-stream'),
    stream;

dotenv.load();

stream = new twitter({
  consumer_key: process.env.TWITTER_CONSUMER_KEY,
  consumer_secret: process.env.TWITTER_CONSUMER_SECRET,
  token: process.env.TWITTER_TOKEN,
  token_secret: process.env.TWITTER_TOKEN_SECRET
});

server.listen(process.env.PORT || 5000);

stream.on('tweet', function(tweet) {
  io.emit('tweet', tweet);
});

stream.connect();

The main reason for proxying the data was to reduce the amount sent out to the browsers, so now was time to take those massive responses and reduce them to some word lists. Again I found a couple of great npm modules to help with this; keyword-extractor for extracting the important words (or more accurately, excluding the non-important words), and franc for determining the language of the tweet (keyword-extractor only works with english, much like my brain).

While writing this I noticed that the twitter response actually contains a lang field, negating the need to use franc. I hadn’t noticed this at the time, oh well!

Plugging these in, along with some exclusions myself (links, retweets, replies) gives us the final code that was deployed to Heroku:

var app = require('express')(),
    dotenv = require('dotenv'),
    server = require('http').Server(app),
    io = require('socket.io')(server),
    xt = require('keyword-extractor'),
    franc = require('franc'),
    twitter = require('./twitter-stream'),
    stream;

dotenv.load();

stream = new twitter({
  consumer_key: process.env.TWITTER_CONSUMER_KEY,
  consumer_secret: process.env.TWITTER_CONSUMER_SECRET,
  token: process.env.TWITTER_TOKEN,
  token_secret: process.env.TWITTER_TOKEN_SECRET
});

io.set('origins', '*:*');

server.listen(process.env.PORT || 5000);

function exceptions(word){
  if (word.match(/https?:/)) return false; // links
  if (word.match(/^@/)) return false; // replies
  if (word.match(/&|\/|"/)) return false; // random punctuation

  return true;
}


stream.on('tweet', function(tweet) {

  // ignore retwets
  if (tweet.retweeted_status || tweet.text.match(/^RT/)) return;

  // only english for now
  if (franc(tweet.text) != 'eng') return;

  // parse that tweet, extract words
  words = xt.extract(tweet.text,{
    language:"english",
    remove_digits: true,
    return_changed_case:true
  }).filter(exceptions);

  if (words.length > 0) io.emit('tweet', words);
});

stream.connect();

So with less than 50 lines of code we have live tweets being parsed for words and those word lists being sent out to the browser. Now let’s get the browser to render them.

This is going to be almost entirely javascript powered so I’m going to concentrate on that, if you’re interested in the HTML and CSS then take a look at the source and ask me any questions you might have.

Firstly we’ll use socket.io to connect to the web socket and start grabbing the words as they come in.

I’m using the underscore.js library here to get access to some simple helper functions

var socket = io.connect('wss://twitter-word-stream.herokuapp.com/');

socket.on('tweet', function (data) {
  _.each(data, function(word) {
    console.log(word);
  });
});

And there we go, the words are being spat out to the browser’s console, but of course this is of no practical use. Lets count the occurences and displaying that visually. We’ll do this by throwing the words and their counts in to an object and then displaying the most popular ones periodically.

var socket = io.connect('wss://twitter-word-stream.herokuapp.com/'),
    word_counts = {},
    text_nodes = {},
    frame = 0;

function render() {
  var max = 0,
      displayed_words = [];

  // increment frame counter
  frame++;

  _.each(word_counts, function(count) {
    if (count > max) max = count;
  });

  // filter them to just the most popular ones
  displayed_words = _.sortBy(_.keys(word_counts), function(word) {
    return max - word_counts[word];
  }).slice(0,30);

  _.each(displayed_words, function(word) {
    var size = words[word] / max,
        text, node;

    // create the text node if need be
    if (!text_nodes[word]) {
      text = document.createTextNode(word);
      node = document.createElement('span');

      // position kind of in the middle somewhere
      var top = 80*Math.random();
      var left = 70*Math.random();

      // give it a random pastelly colour
      node.setAttribute('style', "top: " + top + "%; left: " + left + '%; color: hsla('+360*Math.random()+',50%,50%,0.75)');

      node.appendChild(text);
      document.body.appendChild(node);
      text_nodes[word] = {
        updated: frame,
        node: node
      };
    } else {
      text_nodes[word].updated = frame;
    }

    // clear expired words
    _.each(text_nodes, function(obj, word) {
      if (obj.updated < frame) {
        obj.node.remove();
        delete text_nodes[word];
      }
    });

    // size it relative to it's occurence
    text_nodes[word].node.style.transform = 'scale(' + (0.2 + size*0.8) + ')';
    text_nodes[word].node.style.webkitTransform = 'scale(' + (0.2 + size*0.8) + ')';

  });

}

setInterval(render, 500);

socket.on('tweet', function (data) {
  _.each(data, function(word) {
    word_counts[word] = (word_counts[word] || 0) + 1;
  });
});

There’s a few things to explain here:

This works great, but it will count occurences since you first loaded the page, I wanted it to only consider the most recent words (lets say only the last 5 minutes) so I need to store the word lists in such a way that I can easily and quickly remove the older ones. I could have stored the time of each occurence of each word but that would get complicated. I decided instead to store the word occurences in several different objects (I called them buckets), with the one that was incremented being rotated every few seconds. The render method would then only use the buckets covering the last 5 minutes worth of occurences.

var socket = io.connect('wss://twitter-word-stream.herokuapp.com/'),

    text_nodes = {},
    frame = 0,

    current_bucket = {},
    buckets = [current_bucket],

    bucket_count = 30, // how many buckets to remember
    bucket_width = 10; // how many seconds worth of words to keep in the buckets

function render() {
  var max = 0,
      words = {},
      displayed_words = [];

  // increment frame counter
  frame++;

  // get counts of words across all buckets
  _.each(buckets, function(bucket){
    _.each(bucket, function(count, word) {
      words[word] = (words[word] || 0) + count;
      if (count > max) max = count;
    });
  });

  // filter them to just the most popular ones

  displayed_words = _.sortBy(_.keys(words), function(word) {
    return max - words[word];
  }).slice(0,30);

  _.each(displayed_words, function(word) {
    var size = words[word] / max,
        text, node;

    // create the text node if need be
    if (!text_nodes[word]) {
      text = document.createTextNode(word);
      node = document.createElement('span');

      // position kind of in the middle somewhere
      var top = 80*Math.random();
      var left = 70*Math.random();

      // give it a random pastelly colour
      node.setAttribute('style', "top: " + top + "%; left: " + left + '%; color: hsla('+360*Math.random()+',50%,50%,0.75)');

      node.appendChild(text);
      document.body.appendChild(node);
      text_nodes[word] = {
        updated: frame,
        node: node
      };
    } else {
      text_nodes[word].updated = frame;
    }

    // clear expired words
    _.each(text_nodes, function(obj, word) {
      if (obj.updated < frame) {
        obj.node.remove();
        delete text_nodes[word];
      }
    });

    // size it relative to it's occurence
    text_nodes[word].node.style.transform = 'scale(' + (0.2 + size*0.8) + ')';
    text_nodes[word].node.style.webkitTransform = 'scale(' + (0.2 + size*0.8) + ')';

  });

}

function rotate_buckets() {

  current_bucket = {};
  buckets.push(current_bucket);

  while (buckets.length >= bucket_count) buckets.shift();

}

setInterval(rotate_buckets, bucket_width*1000);
setInterval(render, 500);

socket.on('tweet', function (data) {
  _.each(data, function(word) {
    current_bucket[word] = (current_bucket[word] || 0) + 1;
  });
});

And there we have the (more or less) finished code, and here it is running:

There’s still a few things I’d like to improve when I can:

It was fun to spend a couple of hours playing around with some new things, everyone needs to be able to do that occasionally.

What new technologies are you most excited about playing with?

Photo of Marc Roberts, who wrote this blog post

Marc Roberts is Principal & Co-Founder at Neutron Creations, where he rules over all web development and technical direction. His favourite drink is a sweet manhattan and his favourite sandwich is a croque-monsieur. For a plethora of miscellany follow Marc Roberts on twitter.

comments powered by Disqus