kml_filter.py
Hello,
I am attempting to use the provided kml_filter.py script, but there are some errors that are preventing the script from producing any output:
Any thoughts?
I am attempting to use the provided kml_filter.py script, but there are some errors that are preventing the script from producing any output:
Code: Select all
PS C:\Python27> python .\kml_filter.py -o python_sorted.kml .\WigleWifi_20190317222843.kml
No handlers could be found for logger "fastkml.config"
.\WigleWifi_20190317222843.kml: 1
<generator object features at 0x0000000004384048>
Traceback (most recent call last):
File ".\kml_filter.py", line 193, in <module>
converter.main()
File ".\kml_filter.py", line 70, in main
self.parse(filename)
File ".\kml_filter.py", line 144, in parse
if "Encryption: None" in placemark.description:
TypeError: argument of type 'NoneType' is not iterable
what's the filesize of the input file?
if you load it in google earth, what do you see?
Cheers, and thanks for your interest!
if you load it in google earth, what do you see?
Cheers, and thanks for your interest!
Hi,
The size of the .kml is 8,021KB, 54,859 lines, according to the WiGLE uploads page. This is the transaction ID: 20190323-00024.
Loading the .kml into Google Earth works and I see all the markers, though it brings the app to its knees and hardly functions. I do not see how to easily sort by encryption types in Google Earth either, hence my interest in the kml_filter.py script.
As a side note, I noticed looking at the .sqlite database and the CSV that there is no columnar data for whether or not a network is open? I would think that whether or not a network has encrypted password protection would be an important data point of interest, just as much as the encryption type (WEP/WPA etc.).
Thanks for your quick reply and help.
The size of the .kml is 8,021KB, 54,859 lines, according to the WiGLE uploads page. This is the transaction ID: 20190323-00024.
Loading the .kml into Google Earth works and I see all the markers, though it brings the app to its knees and hardly functions. I do not see how to easily sort by encryption types in Google Earth either, hence my interest in the kml_filter.py script.
As a side note, I noticed looking at the .sqlite database and the CSV that there is no columnar data for whether or not a network is open? I would think that whether or not a network has encrypted password protection would be an important data point of interest, just as much as the encryption type (WEP/WPA etc.).
Thanks for your quick reply and help.
interesting - I gave it a try in my virtual environment and I get a successful run with your file! (yeah, I'm an admin, I can do that)
try blowing away your virtual environment/re-creating, and I'll do the same?
Cheers,
-ark
try blowing away your virtual environment/re-creating, and I'll do the same?
Cheers,
-ark
Trying now in Debian, Python version 2.7.13:
root@debian:/home/laptop/Downloads# python kml_filter.py -o python_filtered.txt WigleWifi_20190317222843.kml
WigleWifi_20190317222843.kml: 1
<generator object features at 0x7f24f4aca370>
Traceback (most recent call last):
File "kml_filter.py", line 193, in <module>
converter.main()
File "kml_filter.py", line 70, in main
self.parse(filename)
File "kml_filter.py", line 144, in parse
if "Encryption: None" in placemark.description:
TypeError: argument of type 'NoneType' is not iterable
Same error.
root@debian:/home/laptop/Downloads# python kml_filter.py -o python_filtered.txt WigleWifi_20190317222843.kml
WigleWifi_20190317222843.kml: 1
<generator object features at 0x7f24f4aca370>
Traceback (most recent call last):
File "kml_filter.py", line 193, in <module>
converter.main()
File "kml_filter.py", line 70, in main
self.parse(filename)
File "kml_filter.py", line 144, in parse
if "Encryption: None" in placemark.description:
TypeError: argument of type 'NoneType' is not iterable
Same error.
question: how are you accessing these KML files? (how/where are you downloading them?)
specifically, the filter tool was written to process the KML format generated by the "link" and "view" tools in the "Uploads" tab of the app, or by downloading using the transid link on the website uploads page; the export run as KML button on the database tab uses a different format.
I see. I might actually be trying to use the the .kml produced by the 'Export' feature from the app. I will try the script again, but this time against the .kml downloaded from the Uploads links here on WiGLE.net
awesome - we actually should evaluate the differences between the formats - these grew organically, but there probably isn't a great reason for the difference.
Cheers,
-ark
Cheers,
-ark
@arkasha your assumption was correct, I was using the .kml produced by and exported from the WiGLE Android app and that was the issue. After downloading the .kml from the Uploads page here on WiGLE.net, the kml_filter.py script ran correctly. Here was my output:
unencrypted: 23
unknown: 2355
wep: 214
wpa: 17179
wpa2: 17119
wpa3: 0
2 questions: first, I can see that there were 23 unencrypted networks according to the script's output, but looking at the .sqlite and CSV exports, there appears to be no columnar data or indication of a network being unencrypted? As I need to run SQL queries against this data set, it would be helpful to know where unencrypted networks are indicated. Second question, is it a bug that wpa and wpa2 have the same number of results?
unencrypted: 23
unknown: 2355
wep: 214
wpa: 17179
wpa2: 17119
wpa3: 0
2 questions: first, I can see that there were 23 unencrypted networks according to the script's output, but looking at the .sqlite and CSV exports, there appears to be no columnar data or indication of a network being unencrypted? As I need to run SQL queries against this data set, it would be helpful to know where unencrypted networks are indicated. Second question, is it a bug that wpa and wpa2 have the same number of results?
1.) the lack of any of those other encryption indicators is the "unencrypted" status. I can't urge you strongly enough to work with the CSV rather than the KML, since the KML just a quick visualization hack on top of the CSV data, and isn't structured for database-style access (the info is just shoved into the "bubble")
2.) that's doing string matching - it may be a bug, or all the networks in your file may contain both strings (usually, the capabilities array contains [WPA],[WPA2] if it's a "WPA2" network.
Cheers,
-ark and the WiGLE team
2.) that's doing string matching - it may be a bug, or all the networks in your file may contain both strings (usually, the capabilities array contains [WPA],[WPA2] if it's a "WPA2" network.
Cheers,
-ark and the WiGLE team
@arkasha in regards to your response #1, that was actually the confusing part for me. For instance, when I examine the .kml for a network called Phillips5G-Guest, I see this:
There is a clear indicator that no encryption is being used. However, looking at the same network in the CSV:
52:6a:03:af:d9:7b Phillips5G-Guest [WPA2-PSK-CCMP+TKIP][WPA-PSK-CCMP+TKIP][ESS]
The AuthMode column shows that WPA2 is being used, and there seems to be no indication that no encryption is being used. The same goes for the .sqlite, there is no indication that the network is open or unencrypted, or Encryption: None.
Code: Select all
Encryption: None
Time: 2019-03-16T15:15:51.000-07:00
Signal: -81.0
Accuracy: 4.0
</description>
<styleUrl>#highConfidence</styleUrl>
<Point>
<coordinates>-111.90319061,40.31055069</coordinates>
</Point>
</Placemark>
<Placemark>
<name>Phillips5G-Guest</name>
<open>1</open>
<description>Network ID: 52:6A:03:AF:D9:7B
52:6a:03:af:d9:7b Phillips5G-Guest [WPA2-PSK-CCMP+TKIP][WPA-PSK-CCMP+TKIP][ESS]
The AuthMode column shows that WPA2 is being used, and there seems to be no indication that no encryption is being used. The same goes for the .sqlite, there is no indication that the network is open or unencrypted, or Encryption: None.
Without being to produce the exact circumstances, I'd recommend using the SQLite or CSV files locally, or the query API online for your purposes; the KML files are intended for visualization rather definitive database-like access, and it's possible that they failed to capture the status of this network correctly. Running spot checks on my own KML exports, I see an Encryption status inside each block.
The snippet included in your example, you seem to have included the tail-end of the previous placemark's description (ending with) and the beginning of the Phillips5G-Guest placemark - however we stop at the end of network ID - this would be followed by a newline and then the Encryption status for that network in the uploads page export format.
Code: Select all
<description>...</description>
The snippet included in your example, you seem to have included the tail-end of the previous placemark's description (ending with
Code: Select all
</Point> </Placemark>
on-site KML export has been updated, as has the kml_filter script to bring it into alignment with client export. We'll update client KML export further in the next release in order to include encryption status as well as BT and Cell export. They'll still contain some disparate fields - there are elements of data that only exist on the client or server - but they'll be parsable by the same tools from now on.
Thanks for the heads-up.
Thanks for the heads-up.
@arkasha That is great to hear that those changes will be made. After looking at the data, I believe the results for WPA and WPA2 are indeed bugged in the script. I used this SQL (querying the .sqlite exported from the Android WiGLE client) to get the proper totals for WPA vs WPA2:
Thank you again.
Code: Select all
SELECT *
FROM network
WHERE capabilities LIKE '[WPA-%';
Who is online
Users browsing this forum: No registered users and 4 guests