Anatum Field Solutions (AFS) is a business that rents Bluetooth GNSS receivers and mobile devices (eg. iPads) for clients interested in collecting high-precision GNSS data. It rents GNSS receivers from a variety of manufacturers. We consider it our responsibility to offer vendor-independent advice on the performance of Bluetooth GNSS receivers by providing information and examples of how the various devices we rent perform in real-world situations. To that end, AFS tests and uses many Bluetooth GNSS receivers to determine how each receiver may best meet its clients’ needs. With years of experience in the field using GNSS receivers, we have a solid understanding of how GNSS receivers perform in the wide open sky as well as difficult environments like heavily-forested areas. Today, there are three popular Bluetooth sub-meter receivers on the market and another in which the manufacturer claims 1-meter accuracy. With all these options and many clients asking us which fits best with their requirements, we ran a number of Bluetooth GNSS receivers on a GNSS Test Course here in Oregon. This testing course has Reference Points (RPs) with known locations, established by a land surveyor using a total station, that range from open-sky to dense tree canopy. This article examines the GNSS receivers we tested and presents their accuracy and precision results.
This article is very detailed to ensure the ability for others to completely understand every step of these field tests and even repeat this test if desired. For readers only interested in results, you can skip down to the “Results” and “Final Thoughts” sections.
GNSS RECEIVERS TESTED
The GNSS receivers we tested included the Bad Elf GNSS Surveyor, EOS Arrow 100, Geneq iSXBlue II, Trimble R1, and Trimble Leap. Since many users are looking to transition to a tablet or smartphone-based Mobile GIS from a handheld GNSS unit and we have rental units on hand, we decided to also run a Trimble GeoXH 6000/Floodlight through the course for comparison. Refer to Table 1 for details of each GNSS receiver.
An important fact to keep in mind is that all GNSS manufacturers publish their accuracy specifications based on open-sky testing (ideal laboratory conditions). They also provide a statistical value defining the accuracy of the receiver. The statistical values are based on horizontal accuracy distributions of 68% (RMS), 95% (2DRMS), or 98% (3DRMS) depending on the manufacturer’s preference. A thorough examination of the statistical definitions can be found in this article from GPS World: Update: GNSS Accuracy: Lies, Damn Lies, and Statistics. It’s important to understand this terminology in order to interpret the manufacturer’s specification.
For example, Table 1 shows the Eos Arrow 100 datasheet specifies 60cm @ 2DRMS (95% confidence) while the Trimble R1 specifies “< 100cm” HRMS (68% confidence), which means the Trimble R1 2DRMS (95% confidence) accuracy is < 100cm x 1.62 = < 1.62 meters. Further, the Trimble Leap, which is marketed as a “sub-meter” receiver, has a stated accuracy of 1 – 3 meters 95% of the time using SBAS (WAAS). So, this is something to keep in mind when researching receivers. Another thing to consider is that not all GNSS interface apps are equal, not only do some have a much more appealing user interface architecture, but each manufacturer implements methods of providing real-time estimated accuracy based on information coming from the satellites. There is not a standard method for estimating accuracy, so some receivers are better at estimating accuracy than others. A later blog post will examine how well each of these receivers estimated accuracy reflected the true horizontal accuracy compared to the known reference point.
Table 1. GNSS Receiver Specs
| Receiver | Rated Real-time Accuracy | Satellites | OS |
|---|---|---|---|
| Bad Elf Surveyor | ~1-meter stationary accuracy with SBAS+PPP | GPS; Glonass | iOS |
| EOS Arrow 100 | 60cm; 95% real-time with SBAS corrections | GPS; Glonass; Galileo; and BeiDou tested with 3 constellations (updated 2017 that the Arrow 100 will use all 4 constellations simultaneously, but not part of this original field test. Only ) | iOS, Android, Windows (7.x, 8.x, 10.x),Windows Mobile/WEHH |
| Geneq iSXBlue II | 60cm; 95% real-time with SBAS corrections | GPS; Glonass | iOS, Android, Windows (7.x, 8.x, 10.x),Windows Mobile/WEHH |
| Trimble Leap | 1-3 meters (95%) | GPS; Glonass | Android |
| Trimble R1 | <1 meter; 68% real-time with SBAS corrections | GPS; Glonass; Galileo; BeiDou(can only load 2 constellations simultaneously) | iOS, Android, Windows (7.x, 8.x, 10.x),Windows Mobile/WEHH |
| Trimble GeoXH | <1 meter; 68% real-time with SBAS corrections | GPS; Glonass | Windows Mobile |
For our testing, all GNSS receivers had SBAS (WAAS) activated for comparison of accuracy from real-time SBAS satellites corrections. Each receiver was tested for both horizontal accuracy and precision. Accuracy is a fairly simple thing; how close was a recorded point to the known point. Precision is the consistency of repeated measurements. Thus, for collecting spatial data, the best would be high accuracy and high precision, but outside of an RTK GNSS receiver acquiring live corrections from an RTK network or from your own RTK base station, that can be difficult to achieve. See our previous article on Accuracy and Precision Related to Surveying.
*NOTE: Out of the box the manual for the Trimble Leap stated its Android and iOS compatible. However, to achieve the Bluetooth link from the Leap to a tablet, the user must utilize the Leap’s app. The situation is similar with the Trimble R1. All the other GNSS receivers are simple to connect in the tablet’s Settings for Bluetooth and can work without also activating the receiver in its respective app. While we were able to achieve a connection with all the other receivers, we couldn’t get the Leap to work with our Field iPads. After a phone call to Trimble, it became apparent that the Leap was not (yet) compatible with iOS. Next, we tried to link the Leap with our brand new rugged Android tablet. This also proved fruitless as the Leap only works with Android phones and not the version of Android running on many Android tablets. At that point, we decided to remove the Leap from our testing group.
THE GNSS TEST COURSE
The GNSS test course has 13 RPs located in Tigard, Oregon. The RPs include locations with open-sky, light canopy, moderate canopy, heavy canopy, and even difficult dense canopy points located within arms reach of 30 to 35-inch Douglas fir trees (See Table 2). The open-sky RPs were originally established using an L1/L2 GNSS receiver and processed through OPUS (post-processing) and repeated a second time to verify accuracy. The remaining RPs were established by a land surveyor utilizing a surveying total station. RP coordinates were recorded in NAD83 (2011) / Oregon North (international ft). ITRF08 coordinates were also computed.
Table 2. List of Reference Points and Canopy Cover Rating
Open-Sky
Light Canopy
| Moderate Canopy
| Heavy Canopy
Dense Canopy
|
The determination of whether an RP should be considered moderate canopy or heavy is strictly subjective. However, since the GNSS test course has RPs with a range of canopy density, we felt it was acceptable to qualify each point with a relative canopy density rating. For reader reference, we have included aerial images of each RP type collected with a UAV. The flight height for each image is approximately 100 feet. Also included is a photo standing on the RP and facing upwards to give a reference for the canopy cover from the ground.
Aerial imagery of each sample location at time of study
TEST METHODS
The challenge in conducting a credible test is standardization and repeatability. We designed the field test to include enough details describing our methodology to enable anyone to repeat this test. Also, keep in mind that some of the GNSS receivers we tested had external antennas (EOS Arrow 100, Geneq iSXBlue II, and Trimble R1 [with an external antenna attached]) and some did not (Bad Elf Surveyor, Trimble R1 [without the external antenna attached)] and Trimble GeoXH).
Receiver Setups:
All receivers were updated with the most current firmware at the time of testing. To accomplish as fair a test as possible, each GNSS receiver was allowed to track satellites for 15 minutes before beginning the course. This actually gave an advantage to the Bad Elf GNSS Surveyor and Trimble R1 because both manufacturers state that to achieve the stated level of accuracy, the receiver must stay stationary for several minutes.
We placed each GNSS receiver antenna on a 2-meter range pole to keep them elevated above our heads and ensure the greatest amount of sky (i.e. most satellites) was visible to each receiver. The Bad Elf GNSS Surveyor does not have an external antenna, so we placed the receiver at the top of a range pole. The Trimble R1 can be used with or without an external antenna. Trimble markets the device as a receiver that users can place in their shirt front pocket without an external antenna, so we tested it both with an external antenna and without. After an initial examination of poor results with the R1 placed in a shirt pocket, we opted for attaching the R1 receiver to a range pole with the unit standing on end to give it a fair comparison to the other receivers (refer to photo).
Standardization:
To further standardize these tests, the user holding the range pole stood at the same compass bearing to each RP for each receiver. In another attempt to minimize any potential user bias, we were joined on-site by a Chief GIS staff member of Ecology & Environment, Inc. Utilizing these methods created the most consistent and fair GNSS receiver field test possible.
Data Capture:
For capturing data, each GNSS receiver was connected to a dedicated iPad running iCMTGIS Pro software and data was collected in ITRF08 (WAAS defacto datum). This both sped up the data collection and minimized any chance of mislabelling data. The Trimble GeoXH collected data in ITRF08 using ArcPad 10.2.3. Each GNSS receiver was run through the test course four separate times at the same time of day. The test required multiple days to complete. To maintain similar testing conditions and results, all testing was conducted between 9 am and 11 am local time so that all receivers were tested against the same satellite constellation grouping.
At each RP, a dynamic line capturing one point (vertex) per second for a total of five seconds was recorded. The standard method for each GNSS receiver was to step up to the RP and bubble level the range pole on the point; count to five; then record a line for five seconds; then move to the next RP location. For the Trimble GeoXH, one user held a 2-meter range pole for the GeoXH to be placed at the correct antenna height and location before counting to five and then recording a line for five seconds.
Analyses:
Data analysis was completed with ArcGIS 10.3. A geodatabase was created containing all the test data converted to NAD83 (2011) / Oregon North (international ft) to match the RPs known location coordinates. Viewing the line data in this way allows for a qualitative examination of precision for all the GNSS receivers across the various environmental conditions on the test course. Finally, each data line was averaged to obtain an averaged data point similar to point-averaging in the field for comparison among GNSS receivers.
RESULTS
Currently, many mobile GIS apps collect points by a single point collect method and lack the ability to average a point. With ArcPad and iCMTGIS Pro, the user can collect a single point or set it for averaging the point collected over a specified time range.
For example, are mobile GIS apps collecting a point when the user hits the “add/collect” button or when they tap the save button after naming the point? Or maybe the app averages the point by starting averaging the point when the user taps the “add/collect” button and then finishes averaging when the user completes the data form and taps the save button? Because of this uncertainty, it becomes even more important to examine the precision of these receivers. If your mobile GIS app only collects a point and doesn’t average, then consider that the point could be captured at any place along the lines shown in the test plots in the Precision Examination section below. If your app averages your point collection, then the result will be better on average and similar to the mean center accuracy results in Table 3 in the Accuracy Results section.
For these reasons, if there are two GNSS receivers that are equally accurate across the tests but one has much better precision, then that receiver is much more desirable.
Precision Examination
The light canopy Valve Cover showed similar precision among all receivers except the R1 and Bad Elf GNSS Surveyor. The Bad Elf Surveyor and R1 with external antenna showed similar precision that was less precise than the Arrow 100, iSXBlue II, and GeoXH. The R1 with the internal antenna has an outlier with massive drift. All outliers in this test were included since it was real world and could implicate a user’s ability to collect field data with confidence.
The four moderate canopy RPs at the Bridge showed similar results at each of the four points. Provided is the plot for the NE Bridge point. The iSXBlue II showed a little better precision than the Arrow 100 which showed better precision than the remaining receivers. The R1 (with and without an external antenna attached), Bad Elf Surveyor, and GeoXH all showed similar mixtures of lower and higher precision with the GeoXH having one outlier with extreme drift.
The difficulty of the environment at the heavy canopy RP at the Manhole caused lower precision in all receivers. The best precision appears to be edged slightly by the Arrow 100 and iSXBlue II. The R1 results both with- and without-antenna attached showed some of the highest accuracies among the receivers at this RP, but the mostly lower precision with some sample points having some of the better precision at this location.
The toughest RPs on the course are the four Root points under dense canopy and placed in close proximity to Douglas Fir trees. The plot of Root 1 shows how difficult these RPs were for the tested receivers. The best precision belonged to the iSXBlue II, followed by the Arrow 100, then the other receivers all had, at least, one short line (high precision) and then two or three lines with low precision.
Accuracy Results
To compare accuracy results, we conducted a mean center analysis of each collected data line to determine an average distance from the data line to the known RP. Basically, it took each data line and determined an average point of its vertices for the line and then we calculated the distance from that point to the known RP and labeled these results as “Accuracy” in Table 3. This method also replicated the point averaging function of ArcPad or iCMTGIS Pro if that had been conducted in the field.
To further examine the precision of each line, we calculated the furthest spread of points for each line and labeled that as “Precision” in Table 3. The spread was not the total length of the data line, but the furthest distance between vertices of each line. The smaller the spread, the tighter the precision. The larger the spread, the worst the precision of that line. For a full examination of all the results, see Table 3.
Table 3. Results of Mean Center Analysis of collected lines
Worst Performing GNSS Receivers
Across all the GNSS receivers, including the Trimble GeoXH, those without external antennas performed less accurately on all RPs and canopy cover types. This was evident in that the Bad Elf GNSS Surveyor and Trimble R1 without-antenna attached had the least desirable accuracy and precision across most RPs. The GeoXH also suffered from low precision compared to those GNSS receivers with external antennas.
Best Performing GNSS Receivers
By far, the best-performing receivers in both accuracy and precision were the Arrow 100 and iSXBlue II. Across all RPs, these two receivers achieved the best accuracy at 11 of the 13 locations. Next, the R1 with-antenna attached had the best accuracy at the two remaining RPs. The GeoXH had nearly the same best accuracy as the Arrow 100 at the open-sky Aluminum Nail location, therefore, both receivers were categorized as most accurate at that RP. The Arrow 100 and iSXBlue II also had the best precision and nearly identical precision ratings at most RPs, with both receivers having or sharing the best precision at all 13 locations.
For examination of the full data set in Table 3, at each RP, the receiver with the best overall accuracy across the four collection points is highlighted in yellow. The receiver or receivers with the best precision (lowest spread) are highlighted green. The receivers with the worst accuracy or precision for the RP are highlighted salmon/pink. In examining precision with the spread, some RP locations had two receivers with nearly identical best or worst accuracy. Therefore, some RPs in the table have more than one receiver highlighted with the “Worst Precision” category (see NE Bridge for example).
FINAL THOUGHTS
These field tests and analyses required a large investment in time and AFS would like to thank Jeff Comstock from Ecology & Environment, Inc., for volunteering his time and assisting with our field tests. Jeff’s efforts helped to provide an outside oversight to ensure an unbiased field test. The test results are provided above for readers to draw their own conclusions to which receiver they believe will work best for themselves and their company, but it should be noted that an external antenna is needed to capture field data as accurately and precisely as possible. There were drastic differences between the quality of data from those GNSS receivers with external antennas and those without. Professionals new to using external antennas with their GNSS receivers in the field may be wondering about the workflow and how to hold the antenna. You can see examples of the various options in our earlier article What to do With a GNSS Antenna?
In order to determine the receiver that best fits the client needs, we’ve defined several criteria for evaluating submeter Bluetooth GNSS receivers:
Accuracy and consistency (precision). Review results from Tables 1 and 3 to understand accuracy and the manufacturer’s claim (eg. RMS, 2DRMS). In our testing, there were significant performance differences among “submeter” receivers, especially under a tree canopy. In our tests, the Eos Arrow 100 and iSXBlue II GNSS consistently performed the rest with respect to accuracy and precision, followed by R1 w/external antenna, the GeoXH 6000, then the R1 w/internal antenna and Bad Elf GNSS Surveyor.
Companion apps. The quality and availability of the receiver’s companion apps on iOS, Android, and Windows varies widely for each receiver and can make a substantial difference in the user experience and quality of data collected. While the Bad Elf GNSS Surveyor didn’t fare well in the accuracy/precision test, it has a beautiful iOS companion app (although it doesn’t work with Android or Windows). The Eos Arrow 100 companion apps offer the best functionality and it has apps for iOS, Android, and Windows. The iSXBlue II GNSS iOS and Windows companion apps have limited functionality and there is no Android app. We rated the R1 iOS and Android companion apps the lowest because their precision estimates were consistently misleading and functionality was limited.
Ease of Bluetooth pairing with tablets. Bluetooth connectivity is a function of how each receiver has implemented Bluetooth and varies significantly among GNSS receivers. The Bad Elf GNSS Surveyor and Eos Arrow 100 lead the pack with Bluetooth connectivity in terms of ease of connectivity. It’s quick and painless. In terms of functionality, the Bad Elf GNSS Surveyor can connect up to five devices at one time while the Eos Arrow 100 has a 1-kilometer Bluetooth range. Both are advantages. The iSXBlue wasn’t far behind, but it can take a few tries to connect via Bluetooth. The Trimble R1 was the most challenging but was acceptable once we figured out the key sequence.
Support for future satellite constellations. With Galileo starting service this year (2016), having more satellites available has the potential to significantly improve receiver performance under a tree canopy. The Eos Arrow 100 includes Galileo in addition to GPS and Glonass. For the iSXBlue II GNSS, Galileo is available as an optional feature. For the Eos Arrow 100 and iSXBlue II GNSS, you can load up to three constellations so the receiver can track and use GPS, Glonass, and Galileo simultaneously. While the R1 supports Galileo, the user is limited to loading two constellations (eg. GPS and Glonass), so to support Galileo, the user would have to remove Glonass (it wouldn’t make sense to remove GPS). The Bad Elf GNSS Surveyor only supports GPS and Glonass constellations.
Manufacturer technical support. Eos tech support is the most responsive and knowledgeable. In addition to the Arrow 100, they supported us on the iSXBlue II GNSS receiver too, even though it’s not their product because the same team designed both the Arrow 100 and the iSXBlue II GNSS. Needed Bad Elf tech support was minimal but friendly and adequate. Trimble R1 support was challenging. While the answers we received were helpful, we were on hold for more than an hour before we could talk to someone. That happened on two occasions.
Battery life. The BadElf GNSS Surveyor lasts 24 hours on a charge, by far more than the others. Its battery is not field-replaceable. The Eos Arrow 100, iSXBlue II GNSS and Trimble R1 last ~10 hours on a charge. Both the Eos Arrow 100 and iSXBlue GNSS have field-replaceable batteries. The Trimble R1 battery is built-inside the receiver and is not user-replaceable.
Price. The Bad Elf GNSS Surveyor is $599, far less than the other receivers we tested. However, we cannot classify it as a submeter receiver based on our accuracy/precision testing as well as Bad Elf’s specification sheet. The Trimble R1 is $2,595. The iSXBlue II GNSS is $2,895. The Eos Arrow 100 is $2,995.
Item of Note
One final comment on the interface app for the Trimble R1 that we feel is important to mention. Of vital importance to users in the field is to understand how the GNSS receiver is performing (eg. estimated precision) so the user has an indicator of the quality of the data they are collecting. During our testing, we observed that the GNSS Status app always estimated the R1’s precision at less than 1 meter regardless of the RP location and canopy cover type. Even without the antenna attached, the R1 was estimating its precision at less than 1 meter at every site, including under the four Dense Canopy Root RPs. Realizing this, we ran through the course and captured a screenshot of the estimated precision for each of the receivers apps that displayed it, for each RP, including the R1. Not only did R1 GNSS Status always show the precision of less than 1 meter across the entire test course, but it returned estimated precision values of 0.56m, 0.57m, or 0.59m, 42% of the time. In reality, the R1 rarely delivered precision results that it estimated.
Example: RP location NW Bridge. Trimble R1 screenshot on the right showed 0.63 meters estimated precision but the actual error was 3.47 meters. This is a major concern because users rely heavily on the estimated precision provided by the receiver to make decisions on when to store data and then also record the estimated precision as GNSS metadata in their GIS to make decisions at a later date. A quick summary of actual receiver performance vs. estimated is as follows:
Bad Elf GNSS Surveyor – Never over-estimated actual accuracy.
Eos Arrow 100 – Over-estimated actual accuracy 15% of the time.
Geneq iSXBlue II GNSS – Over-estimated actual accuracy 33.3% of the time.
Trimble R1 without external antenna – over-estimated actual accuracy 100% of the time.
Trimble R1 with external antenna – over-estimated actual accuracy 66.7% of the time.
Lastly, with the exception of the Trimble R1 without an external antenna, every receiver’s estimated precision on the open sky RPs did a reasonably good job of accurately estimating the precision. The challenge for most of the receivers was accurately estimating the accuracy under a tree canopy, as which some did a better job than others based on the data in the above tables.
Copyright 2016 Anatum Field Solutions
Tagged Products
