question on query limits

Suggestions for WiGLE/JiGLE/DiGLE

15 posts • Page 1 of 1

Postby ianrose » Thu Apr 30, 2009 11:14 pm

Hi all -

I have been hit with the "too many queries" error page when conducting a query. I understand that there are query limits in place to keep people from overloading the webserver, but I am unable to find any specifics anywhere on the forums or elsewhere on the site.

I have a list of ~4k bssids that I am trying to look up. I am doing so programmatically and have inserted a 1 second sleep between each query to avoid posting queries too quickly, but after ~100 queries I started getting the "too many queries" error nonetheless.

Are there any guidelines on how services should interact properly with wigle.net to avoid this problem? I'm more than happy to write my code to behave nicely but without any documentation on how wigle.net WANTS me to act, I can't do much other than guess...

Thanks for any tips.

Postby whitedice » Fri May 01, 2009 3:38 am

Why are you trying to query 4000 networks?

Postby ianrose » Fri May 01, 2009 3:40 am

Why are you trying to query 4000 networks?
Because I have captured data from 4000 different BSSIDs, and I would like to know where the transmitters are located. Perhaps I don't understand what you are trying to ask...

Postby whitedice » Fri May 01, 2009 4:14 am

That just seems like a daunting task..

What kind of program did you use to collect? I might be able to create a KML file you could open in Google Earth, if you used Netstumbler (*.ns1) or Kismet (*.csv)? That is, if you are willing to send me your file.

Postby ianrose » Fri May 01, 2009 4:17 am

That just seems like a daunting task..

What kind of program did you use to collect? I might be able to create a KML file you could open in Google Earth, if you used Netstumbler (*.ns1) or Kismet (*.csv)? That is, if you are willing to send me your file.
I used a custom-built libpcap-based program. How would you be able to create a KML? Do you have your own database of MAC-to-latlong entries?

Postby whitedice » Fri May 01, 2009 4:39 am

Er... well.. it depends on what kind of information you collected when you ran your program. If you collected points without any related GPS coordinates, then I am of no use. If you are just randomly searching every MAC address you found against Wigle's Database..... then I am of no help. I can try and visualize GPS coordinates a variety of ways. I have been working on a Kismet CSV to KML VB program for my Final Project. I also have some GIS software that I could overlay coordinates on a generic map. I need coordinates of some kind though in a text file or an excel spreadsheet. If you are just randomly searching MAC Addresses to see if they have been located, my only suggestion is to possibly increase the delay between queries, though I couldn't recommend any value.

Postby ianrose » Fri May 01, 2009 4:41 am

Er... well.. it depends on what kind of information you collected when you ran your program. If you collected points without any related GPS coordinates, then I am of no use. If you are just randomly searching every MAC address you found against Wigle's Database..... then I am of no help. I can try and visualize GPS coordinates a variety of ways. I have been working on a Kismet CSV to KML VB program for my Final Project. I also have some GIS software that I could overlay coordinates on a generic map. I need coordinates of some kind though in a text file or an excel spreadsheet. If you are just randomly searching MAC Addresses to see if they have been located, my only suggestion is to possibly increase the delay between queries, though I couldn't recommend any value.
Ah, I understand now. Unfortunately I have no location information (indeed that's what I'm trying to get from wigle.net), just the bssids. Thanks, though, for the offer. 8)

I'm hoping a site admin will wander by and give me an idea of what kind of parameters to use (like delay between queries or something)...

Postby whitedice » Fri May 01, 2009 4:53 am

So there really isn't any guarantee any are in the database? What kind of numbers have you seen before it errors? Another suggestion, depending on how long it took to gather the original data, is to redo/rewalk/redrive the area with Kismet or Netstumbler with GPS, and then compare results? Kismet and Netstumbler are both free. If you are trying to see how well your program works, that may be the best route. I guess it depends on the coverage of the area you are looking in, as well as other circumstances mentioned.

Postby ianrose » Fri May 01, 2009 4:56 am

So there really isn't any guarantee any are in the database? What kind of numbers have you seen before it errors? Another suggestion, depending on how long it took to gather the original data, is to redo/rewalk/redrive the area with Kismet or Netstumbler with GPS, and then compare results? Kismet and Netstumbler are both free. If you are trying to see how well your program works, that may be the best route. I guess it depends on the coverage of the area you are looking in, as well as other circumstances mentioned.
Correct - in fact only about 25-30% of them (so far) have actually been found in the wigle.net database. But I expected that a significant number wouldn't be found - that's ok. I'm glad just to get lat/longs on whatever I can.

Postby whitedice » Fri May 01, 2009 5:14 am

Ugh.. that seems like a lot of unrecorded points you could be getting credit for in the database. If you were doing this near OH, I'd be all over that. If you recorded them with GPS, Wigle would create you a pretty map of the new ones you discovered.

That just seems like a lot of effort for little results. I'm all about mass vacuum collection. It is interesting you created a program that can do what it does. Is this for research of some kind?

Postby ianrose » Fri May 01, 2009 5:39 am

Ugh.. that seems like a lot of unrecorded points you could be getting credit for in the database. If you were doing this near OH, I'd be all over that. If you recorded them with GPS, Wigle would create you a pretty map of the new ones you discovered.

That just seems like a lot of effort for little results. I'm all about mass vacuum collection. It is interesting you created a program that can do what it does. Is this for research of some kind?
Yeah - I'm a graduate student in computer science. The data isn't really uploadable to wigle.net though because I am using faily high-gain omni antennas from fixed locations (I'm not actually wardriving at all). So I don't have much of an idea of each network's location.

Postby uhtu » Fri May 01, 2009 11:41 am

the query limit is far far below 4000 queries/day. that isn't fixable with a sleep() in your script.
contact wigle-admin@wigle.net with what you're up to and we may be able to help.

Postby i_do_dew » Sat May 02, 2009 4:10 am

Yeah - I'm a graduate student in computer science. The data isn't really uploadable to wigle.net though because I am using faily high-gain omni antennas from fixed locations (I'm not actually wardriving at all). So I don't have much of an idea of each network's location.
If you are trying to test how good your hi-gain antenna is working based on the distance between you and the AP's you have found, you would be better served to instead of querying all the points seen, only query the ones that fall below (n)db. Where n is a fairly large negative number. The problem with that method is it assumes that every one of those AP's has the same operating power, antenna orientation and environmental mounting. In reality it wont tell you anything that that helps you determine the gain. The way an antenna gain is compared is against a reference antenna detecting the same source. Normally reference is a 1/4 wave isotropic radiator.

Because such an item can't be built, actual antenna comparisons are done with a 1/4 wave dipole as the reference. In the sad case you dont have the signal strengths from your program, go back and find a way to get that information. and if you are getting such a low hit rate, you may consider wardriving the area first to find the AP's yourself. for all you know, someone may have moved since the AP's were last plotted in wigle. Especially if this is a college area.

Postby stilia.johny » Tue Dec 31, 2019 12:19 pm

Hi folks,

I suppose it is still unknown of how many API calls we get per day ?

Its almost impossible to find this info on any of the FAQ etc

Thanks

Postby arkasha » Wed Jan 01, 2020 2:49 am

It's dynamic based on how likely you are too be human (and not a harvesting bot)

Cheers!

15 posts • Page 1 of 1

Return to “WiGLE Project Suggestions”

Who is online

Users browsing this forum: No registered users and 5 guests