format?

The gear needed for wardriving

7 posts • Page 1 of 1

Postby flok99 » Mon Apr 02, 2018 10:37 am

Hi,

I've created my own wardriving setup (ESP8266 + GPS module and a bunch of LEDs of course).
Now I'm logging the following:

date
time
latitude (degrees)
longitude (degrees)
altitude (m)
mps
# satellites
hdop
bssid
channel
rssi
hidden (0/1)
encryption method
ssid

My question now is: in which format should I upload this data? And is there a file format description and an example availale somewhere?

Postby arkasha » Tue Apr 03, 2018 5:24 am

Hi flok99,

Awesome - we love it folks build new stuff!

If you want a simple reporting format, the CSV files we use in WiGLE WiFi Wardriving are pretty straightforward. The basics are mentioned here. We haven't formally documented it (we should), but since it's self describing, we've been lazy. There are examples in people's test cases, which is good practice!

If you're thinking of going further, we support all but the most recent Kismet file format, and those are *comprehensive* and well thought-out!

Cheers,

-Ark

PS> our lack of documentation is remedied in a very small way by WiGLE WiFi Wardriving being an open-source project - you can read how we do it!

Postby flok99 » Tue Apr 03, 2018 7:31 pm

Thanks!

It looks like I got it to work: the uploads-screen shows detected wifi stations.

I wrote a blog-post about my wardriving device: https://vanheusden.com/misc/blog/2018-04-02.php


Hi flok99,

Awesome - we love it folks build new stuff!

If you want a simple reporting format, the CSV files we use in WiGLE WiFi Wardriving are pretty straightforward. The basics are mentioned here. We haven't formally documented it (we should), but since it's self describing, we've been lazy. There are examples in people's test cases, which is good practice!

If you're thinking of going further, we support all but the most recent Kismet file format, and those are *comprehensive* and well thought-out!

Cheers,

-Ark

PS> our lack of documentation is remedied in a very small way by WiGLE WiFi Wardriving being an open-source project - you can read how we do it!

Postby PhattyMo » Sat Apr 07, 2018 2:06 pm

Hello flok99 and other Wiglers!

This is excellent! I'm quite interested in this project,as I've been playing with the ESP8266 for a couple of years now,and more recently the ESP32 also. These little things are great! I started on a similar ESP8266 Wardriving project a while ago,but ran into a couple of bumps while trying to figure out the code,and never got around to finishing it. (that 'daily life' thing getting in the way.) I'm not much of a coder,I'm more of a hardware guy,so my code is probably horrible,though I can usually,kinda,mostly,figure it out,and make it work.Maybe.

Anyways,after some tinkering with your code,and looking at .CSV files from the Wigle app over the last couple of days,I think I've managed to get the ESP to automatically format the files on the SD card in a 'Wigle-like' .CSV format.
Here's what I've done:
I added a Wigle-like .CSV file header (Copied from your "process-wigle.py" Python code),when the file is first made on the SD card,something like this:

Code: Select all

fh = SD.open(fname, FILE_WRITE); fh.println("WigleWifi-1.4,appRelease=2.27,model=SM-G960Z,release=8.9.0,device=starlte,display=R16NW.G960FXXU1ARC5,board=universal9810,brand=apple"); // Write File header,for a Wigle-like .CSV structure. fh.println("MAC,SSID,AuthMode,FirstSeen,Channel,RSSI,CurrentLatitude,CurrentLongitude,AltitudeMeters,AccuracyMeters,Type");
I also re-ordered some of the data to fit,like so:

Code: Select all

fh.print(WiFi.BSSIDstr(i)); fh.print(','); fh.print(WiFi.SSID(i)); fh.print(","); printEncryptionType(WiFi.encryptionType(i)); // Reformatted to a Wigle-like .CSV structure. fh.print(","); fh.print(year); fh.print('-'); fh.print(month); fh.print('-'); fh.print(day); fh.print(" "); fh.printf("%02d:%02d:%02d.%02d", hour, minute, second, centisec); fh.print(","); fh.print(WiFi.channel(i)); fh.print(","); fh.print(WiFi.RSSI(i)); fh.print(","); fh.print(lat, 6); fh.print(","); fh.print(lng, 6); fh.print(","); fh.print(alt); fh.print(","); fh.print(hdop); fh.print(","); fh.println("WIFI");
And I copied a bit of code from an Arduino example(the "ScanNetworks" sketch),to do the encryption type 'detection',and add it to the file (note the "printEncryptionType" bit,above..it leads to here) I put this at the very bottom of the sketch:

Code: Select all

void printEncryptionType(int thisType) { switch (thisType) { case ENC_TYPE_WEP: //Serial.println(" WEP "); fh.print("[WEP][ESS]"); break; case ENC_TYPE_TKIP: //Serial.println(" WPA "); fh.print("[WPA-PSK-TKIP][ESS]"); break; case ENC_TYPE_CCMP: //Serial.println(" WPA2 "); fh.print("[WPA2-PSK-CCMP][ESS]"); break; case ENC_TYPE_NONE: // Serial.println(" None "); fh.print("[None]"); break; case ENC_TYPE_AUTO: //Serial.println(" Auto "); fh.print("[Auto]"); break; } }
(EDIT: The code got kind of mangled,but hopefully you can still make sense of it.)


Possible tweaks/questions;
1: In the file header part,I'm not sure what would be a better alternative to just copying a random make/model of phone,app version,etc. Perhaps it doesn't matter much? Maybe we could make a new category for "IoT"/ESP/"other" devices? I don't know. I mean,this isn't even a phone,and certainly isn't running the Wigle app. Maybe just 'zero it out' somehow,or leave it blank?

2: The encryption detection/logging bit is kind of basic,and funky,currently. I plan on trying to have that make more sense,and be more like a .CSV from the Wigle app. I just kind of copied some semi-sensible stuff into there,for now,to test if it even remotely works...and it seems it does! The site accepted,and processed some test files I uploaded,and it appears I even found a couple new networks while I was testing/tinkering with it! I also got good looking .KML's back from Wigle. I got so excited I decided to make this post. :D I love this stuff!

Maybe the excellent Wigle devs can share any thoughts they have on the above points? Does kinda wonky data in those fields (File header,and encryption types) matter much? Any suggestions to make it better/more accurate/whatever? There may be some hardware limitations here,as I'm not sure the ESP8266 supports/can distinguish all of the various encryption types. Maybe just the basic WEP/WPA/WPA2/None/Auto are enough?

Other than some random stability issues that may be attributed to the poor connections in my breadboard,this seems to be working fairly well so far for me. Still a bit crashy though. I'm using a 'bare' ESP8266 -07 module w/external 9dBi whip antenna,SD card breakout,and a uBlox NEO-7 GPS module. It catches about 40-50 networks per scan,at my location.
In an attempt to resolve the random stability issue(s) I'm having,I made some hardware changes also. I moved the SD card SS pin to GPIO 16 instead of 15,to avoid conflict with GPIO 15 (D8,on the WeMos) because the ESP8266 is picky about that pin,since it is used for boot-mode selection. I also removed the status/lock LED bits,since it is also picky about GPIO 0 and GPIO 2 (D3 and D4 on WeMos),since these are also related to boot-modes. (plus, I didn't want it to be too 'blinky'.) I also moved the start/stop button to GPIO 2. I've got pull-up resistors on GPIO 0 and 2,along with the buttons,and pull-down on GPIO 15. I also pared down the serial output,thinking that may help some. It's much better than when I first started,but still crashes at random sometimes. Maybe I just need to move it off the breadboard,and actually solder the connections.
I'm not 100% sure how the WeMos modules are setup,with regards to pull-up/pull-down on the pins,etc. I've never used them. I'll have to order a couple,and tinker with them.Maybe that will be more stable for me.
I'd love to get this down to roughly the size of a matchbox,using an ESP8266 -12 module,and power it directly from a small lithium cell of some sort. A tiny wardriving setup!
Even more fun might be doing something similar with the ESP32,since it also includes Bluetooth capability.


Big thanks to flok99 for releasing the code,so I could find it and tinker with it. :) I'm having fun playing with this. I'd love to hear any thoughts/comments/questions/feedback the community has.

Cheers!

Postby arkasha » Sat Apr 07, 2018 5:41 pm

Hi PhattyMo,

Excellent post! Sorry about the forum code formatting - extra newlines can help, but we should really figure out how to get better code formatting into this forum package.

To answer your questions:
1. the app version is effectively the file format version, so keeping that consistent is important. The phone attributes are free-form, and you can and should update those to reflect the device you're using. As we mention here, if you update those values, you'll start appearing in the device statistics (we'll update that page text to indicate more than just phones). "model," "device," and "brand" all correspond to columns on that page. Release and version display also show up there, but won't mean much for non-android device - maybe establishing a simple norm (0.0, OTHER?) for that would be easiest for those?

2. The capabilities sets from android are pretty comprehensive if you need a template; how can we help?

Postby PhattyMo » Sun Apr 08, 2018 5:51 am

Hey arkasha,

I'm glad you like it!
No worries about the code thing,at least it's posted up,and somewhat decipherable,as a reference,or in case anybody else wants to tinker. It shouldn't be too hard to re-format it a bit back to normal.

Okay,so I'll change the appRelease in the header,so it matches the version on my phone that I'm modeling it after. My phone is still running 2.14. (I should probably update it some day.)

My initial thought for the brand/model data in the .CSV header was maybe something like this?

Code: Select all

"WigleWifi-1.4,appRelease=2.14,model=ESP8266,release=0.0,device=ESP8266-07,display=0.0,board=0,brand=Microcontroller"
So that there would be a new 'manufacturer' added,called "Microcontroller",and then the "model" could kind of denote the uC platform;ESP,Arduino,ARM,etc. with maybe some more specifics/part numbers under "device"? That would include a fair amount of info about the specific device. Does this seem acceptable enough to go with? Any thoughts?

For the encryption type stuff,I went looking around the Arduino/ESP8266 reference here: http://arduino-esp8266.readthedocs.io/e ... yptiontype
and it seems it only returns one of those 5 different results,so we might be kind of stuck with those,with no love for other things,like WPS,ESS,or other combos I see in the app .CSV file,like "WPA-EAP-CCMP+TKIP."
I guess I'll just leave it as is,for now,with just the basic WEP/WPA/WPA2/Auto/None encryption types. Hopefully that is acceptable?

There might be a way to get more of that data,if you go 'under' the Arduino layer,and put the ESP8266 into promiscuous mode,and read/decode the raw frames/packets,or something like that? I'm not entirely sure,I think that might be a bit beyond my coding capabilities,at the moment. :) (For anyone interested,This might be a helpful example of doing something similar,to sniff out probe requests. https://github.com/kalanda/esp8266-sniffer )

Postby arkasha » Sat May 02, 2020 6:28 pm

finally documented the CSV file format:

https://api.wigle.net/csvFormat.html

7 posts • Page 1 of 1

Return to “Net Hugging Hardware and Software”

Who is online

Users browsing this forum: EOT and 2 guests