Monthly Archives: November 2015

AltBeacons with Android

I was reading my good friends Juhana’s Blog about BLE BEACONS ON WINDOWS 10 and was wondering on how would I do these beacons with Android.

Juhana did his BLE work alongside my another friend Tomi, thus I of-course did check Tomi’s Blog, and got his example for Advertising & scanning for Altbeacons from his WORKING WITH CONSPICUOUS DEVICES article. This example implementation was then used later on to check that it can see my android advertisements in right way, as well as that my android implementation would see the windows device Altbeacons  as well as could decode the scan record right way.

Juhana had nice illustrator showing the advertisement package used with AltBeacons, and it looks like this:

AltBeaconScanRecord

Juhana also mentioned in his article that “DataType value should be a constant 0xFF as specified in the AltBeacon protocol specification”, and after seeing this I went to check what the Generic Access Profile says for the type 0xFF.

As the 0xFF type means ‘Manufacturer Specific Data’, and it’s the first type on the package, we simply need to get an advertisement scan record, which only has the manufacturer segment and we are done.

With the BluetoothLeAdvertiser API, to take out any unnecessary data from the scan record, we need to:

  • With AdvertiseSettings call setConnectable with false. This makes the beacon not being connectable, also it makes sure the flags part in scan record is not added.
  • With AdvertiseData call both setIncludeDeviceName and setIncludeTxPowerLevel with false value to make sure they are not included in the scan record either

The to set the data for the scan record, we can use the AdvertiseData::addManufacturerData(int manufacturerId, byte[] manufacturerSpecificData) function. Note that the manufacturer id is encoded to the scan record automatically by the API, and it takes 2 bytes. Rest of the data is supplied by byte buffer and its written directly to the scan record, thus you can use your own encoding inside it.

Following picture illustrated how the data is written into the scan record. The Length & Type fields are set by the API and are both taking 1 byte. The manufacturerId is set from the value you supplied with addManufacturerData call and it takes 2 bytes. The green part was also supplied alongside the call for addManufacturerData function, and it can be up to 27 bytes long (31-1-1-2).

AltBeaconScanRecordAndroid

I did also quick search for any library that could be used with altbeacons and did find some. In general they looked a bit too big for the simple task needed here, and I wanted to have code that is really minimalistic.  Also wanted to have codebase that could easily be modified to support different data schemes used for the 27 byte payload.

In general, only codes that would be needed here, are involved on encoding & decoding the scan record data accordingly.

The resulting example is called AltBeacon  and you can find it from github under my BLETestStuff repo 

I implemented both encoding & decoding in the AltBeaconFactory.

Generally the code is rather strait forward, for example the encoding part simply puts the variables stored in AltBeacon  class instance into a byte array according to the type and encoding used in the scan record.

public static  byte[]  getAdvertDataFromBeacon(AltBeacon beacon) {
    byte[] ret = new byte[24];
    //beaconCode
    int beaconCode = Integer.decode("0x"+ beacon.getBeaconCode());
    ret[0] = (byte)((beaconCode >> 8) & 0xff);
    ret[1] = (byte)(beaconCode & 0xff);
    // id1 2 - 17
    String tmpString = beacon.getId1().replaceAll("[\\s\\-]", "");
    int index = 2;
    for(int i = 2; i <= tmpString.length(); i = i + 2){
        if(i <= 32){
            ret[index] = (byte) (Integer.decode("0x" + tmpString.substring((i - 2), i)) & 0xff);
            index++;
        }
    }
    //id2
    int id2 = beacon.getId2();
    ret[18] = (byte)((id2 >> 8) & 0xff);
    ret[19] = (byte)(id2 & 0xff);
    //id3
    int id3 = beacon.getId3();
    ret[20] = (byte)((id3 >> 8) & 0xff);
    ret[21] = (byte)(id3 & 0xff);
    //refRSSI
    ret[22] = (byte)beacon.getRefRSSI();
    ret[23] = (byte)beacon.getManufacturerReserved();
    return ret;
}

The BeaconScanner is implemented in a way that it is using BluetoothLeScanner with devices having Lollipop or never, and with older devices its using the BluetoothAdapter::startLeScan() API.

One notable issue I run into with Marshmallow was the new security scheme google implemented with it. With BLE beacons, when the code is run with device shaving the Android 6.0 (i.e. Marshmallow) you are required to add ACCESS_COARSE_LOCATION into then manifest file.

On top of having the permission defined in the manifest, you also need to ask users permission for the capability. For implementation details check the MainActivity  class implementation.

The code was tested with API level 18 (Jellybean), 19 (KitKat), 21(Lollipop) as well as 23(Marshmallow) devices. Here’s video showing how the app runs on these different devices:

The distance shown on the screen is of course a bit off because I did not have any good way on getting really good value for the measured power sent with the scan record.

But the video does show clearly the fluctuations on the distance calculated, and this is caused by the measured Rssi, thus to get the distance fluctuation out, we should use some kind of averaging the Rssi over time, instead of the last measured value alone.

Thirdly, it can be seen that the values are not really accurate, thus the measured distance should not be shown as meters, instead we should use zoning.

Good example of the zoning can be seen with splitting the distance into three zones with iBeacons:

  • Immediate: Within a few centimeters
  • Near: Within a couple of meters
  • Far: Greater than 10 meters away

 

Quick research on BLE discovery

We have been having the Thali postcard demo ready for quite a long time, and while we been using it, it has been really easy to see that the Wi-Fi Direct discovery is really not up to par. It does work, time to time, but most often it just fails to discover some peers, or occasionally it simply takes way too long it to discover anything. On top of this, it very often also stops seeing peers that are still around, and marks them unavailable, thus making it impossible to communicate with them.

One alternative approach for the discovery would be to use BLE, and as the data we require to exchange can’t be included in the scan record data (it has total limit of 31 bytes), we can’t use simply beacon approach, but instead we would need to use a real connectable peripheral, which would serve the data via the GATT service.

As I was going over this earlier in summer, we did not really have a good view of what hardware requirements BLE would pose. In essence BLE central (i.e. scanning) has been working since API level 18, though some might argue that it’s crappy with API level 18, and you should actually use API level 19.

But with peer discovery, we do also need to be able to advertise our presence, for this we need to get the BLE Peripheral to be supported. This is then supported from API level 21 (Lollipop), and same time there was new scanning API released, bringing better battery usage as well as supposedly tons of improvements. So far with my simple tests, I have not seen any big differences between the BLE scanner APIs, though with KitKat API you do need to create own parser for the scan record.

So from the API point of view the requirement is that the device must support Lollipop. Then we soon discovered that not all Lollipop devices support being BLE Peripheral. For example LG Nexus 5, did have support for the API in early pre-launch versions of the Lollipop for it, but with 5.0.1 at least it was removed. Supposedly the reason being that the HW with it does not support being BLE peripheral & central same time.

For Thali discovery the requirement indeed would be that the device must be able to simultaneously to advertise its presence as well as to seek other devices advertising their presence. Then quick look into the Bluetooth specs made it very clear, it indeed appears that Bluetooth 4.1 lets any device be both a peripheral and a hub at the same time. Small change in terminology there, but indeed that’s what was needed. Thus the revised device specs was that all devices must support Lollipop as well as have Bluetooth 4.1 HW.  Back in the summer when I made my first list, I did only find less than 10 devices which fitted into these specs, luckily the numbers of devices is increasing rapidly.

Then as I now knew which devices I would need to have, the next task was to create a nice set of codes for advertising & discovery, and to do some testing to see how well the API behaves with the intended usage. The results app can be found at Github.

The apps logic got a bit complicated. I was having good amount of issues with BLE connection & characteristics reading, basically when you design your logic, you should remember:

  • Only one request of any kind can be active at any given point
  • If there is an error, you must have a short break before re-attempting anything, otherwise the device might get into a state it cannot recover from the error anymore.
  • With some devices, it appears that you should run all stuff in one thread (UI thread might be preferred)
  • Don’t ever call gatt.close() right after gatt.disconnect(). This will cause internal error inside the API, instead simply call disconnect, and do close inside the onConnectionStateChange callback

And even when remembering all of these you’d end up getting occasional 0x85 / 133 GATT_ERROR errors, which after you should have a break (30-60 seconds should be fine) before you do anything else with BLE.

The logic for the discovery with the example app, is to

  • first find the peer via BLE scanning, and then
  • to do connection to it, which after we
  • request characteristics, find ours and
  • read the characteristics value

Important issue here is that we should always start connection freshly discovered peer. Since if we store a discovered peer for later connections, and we get to try connections after it has moved out of reach, we would be getting connect error, which would require us to have a break on API usage, making out discovery to work slower.

Then with initial tests I had 3 devices meeting the specs doing full discovery rounds once in a minute. And did determine that all worked just fine. There were errors with some connections, but re-tries did succeed eventually and I could run the tests for hours without having problems.

Then to make things more interesting, I did also boot up 5 additional phones which were doing only the discovery (they did not have Bluetooth 4.1, and thus could not run the advertising API), and strait away I started to see more error with the original 3 devices. And after running the tests for several days, I did not find any ways on making the logic to work reliably.

In all tests, after just couple of hours of running, one or two of the 3 devices doing both advertising & discovery, started to have error situations where they never recovered. Most often the connection request to the GATT returned the 0x85 / 133 GATT_ERROR, and I did not find any nice ways I could get the app to recover and start succeeding on the discovery.

The issue appears not to happen if we do less work on BLE, thus I changed the logic this in mind. First thing to do here is to save all discovered devices data, and if we see the same device again, we simply use the old data, which indeed reduces the work we do with BLE.

With android the BLE address is changed every now and then, the actual value depends on the manufacturer, but its guaranteed to change after x seconds. Also the BLE address will be changed every time we stop & start the BLE advertising, thus, if we need devices to re-do the full discovery, we have a nice mechanism for getting it done.

Then I decided that same time as I do read for the characteristic on the other device, I could also do write and give it my discovery data. That way it would not need to connect me to get my data, and thus it would be doing less work on BLE. I’m calling this Discovery Push via BLE.

Then I realized that I actually have some bytes I’m not using with the BLE scan record. I only have the flags there, and the full length UUID. Thus I actually had enough space left for adding 6 byte long Bluetooth address in it. Which would allow me to make insecure Bluetooth connections for exchanging the discovery data.

Thus I revised the advertiser code to include the Bluetooth address as a service data, and in discovery parts I added code that does insecure Bluetooth connection to the advertising device, if the BLE discovery would fail. Effectively also reducing the work we do with BLE.

With insecure Bluetooth connections, I could also do discovery push, similarly as I do with BLE characteristics. The problem here is than that we are using the BLE address as an identifier for the peer. With BLE we can see the address inside the callback that gets called when we issue the write. But with insecure Bluetooth, we of course only see the ‘normal’ Bluetooth address in the receiving end. And as I have not figured a way on reading my own BLE address so far, we couldn’t use this method for pushing discovery, unless we can deal with the lack of BLE address.

The following image illustrates the resulting logic for the discovery.

AndroidBLEDiscoveryFullStuff

With the final logic, at least I could run the discovery codes for hours without any of the devices going to any irreversible bad state, though I only had 3 devices acting the both advertiser & scanning parts, and 5 additional devices doing the scanning. Thus more testing with higher number of devices would be needed before I would trust the discovery code. Anyway, currently I’m not working on thali, thus that work would be needed to get done by somebody else.