Understanding the Communication Protocol of Apple HomeKey

I am currently analyzing the communication protocol of HomeKey. Here are two traces between the lock (reader) and the device (card).

First set:

r2c > 80 80 01 01 6B 5C 02 02 00 87 41 04 34 BC DE 44 04 90 01 8D 24 02 18 2B 80 46 2A D6 25 7D 52 03 64 10 5D 9C 66 5D 20 9A 51 BC 18 DF 23 4A A2 C2 2B 01 2F EC FF 3C 96 D7 82 88 A6 78 75 C2 43 B5 AD B0 00 86 2A 11 9B B6 A3 22 15 8E 4C 10 D9 5F F1 AD F9 5E 35 CD 59 6F 87 A2 AA 3C 4C 6F 4D 10 F0 9D 22 85 B6 58 B8 64 5D 49 E5 BA 7F 0D 3D E3 00

c2r < 86 41 04 F9 54 5B 77 F5 BA 34 5D 0F 0F 20 E7 D7 7F F0 9D F7 5C 06 D6 02 4B AB 48 E5 A4 C8 C1 DC BC CB B7 0F F4 0C 2D A6 29 67 9F CC E7 5A 5A 75 8C 2A 45 D4 51 32 0F F4 D3 E0 7E 5A C8 E7 1E 8D E7 E7 33 9D 10 88 C7 A3 ED A1 7E BE DA 9C F8 74 63 30 55 32 C0 90 00

Second set:

r2c > 80 80 01 01 6b 5c 02 02 00 87 41 04 94 2a c9 ca a3 72 f2 6a 5d 27 55 c2 53 57 fc 86 4f db 59 d9 20 1d f0 4f f7 30 f3 42 05 12 e7 9e 36 06 b2 cf 19 5c c8 63 20 41 6e 85 5e 12 72 e8 c8 15 c0 3e 2c 30 b9 a7 5c 96 3a b0 3a 45 da c8 4c 10 73 8d 2a 47 f1 59 3c d1 3b 80 e6 cb 53 d4 30 c5 4d 10 f0 9d 22 85 b6 58 b8 64 5d 49 e5 ba 7f 0d 3d e3 00
c2r < 86 41 04 3A 24 6E D0 1F DE 67 3F 1C 3F 90 71 E7 C0 F7 B3 8E 36 5E 84 77 40 05 3F 2A 00 5E 6B DD E0 71 B9 2B 54 BF 07 07 BC 98 69 6C 72 C0 E9 93 0C C1 BA 6F A5 78 B7 3E A2 82 47 03 24 C0 AB 96 7D 80 83 9D 10 4B 53 6F 7D BE 00 BA DC A8 55 40 BD 2E 21 51 B8 90 00 

I suspect that the ECDSA is being used here. The reader first sends a public key (maybe) and two random numbers and the card then responds with another public key (or signature) and a number (but should not be a random one).

What are possible protocols that may be applied here? I am really confused.

Reverse engineering! These sorts of things can be fun. I know from experience. You are actually doing pretty good so far. I think your suspicions about them using ECDSA are likely accurate.

The frames you are showing above are called application protocol data units (APDUs). APDUs have requests and responses. In the request, the first several bytes (variable length, of course) define various aspects of the APDU. Once you decode those you can then work on trying to interpret the payload.

The response APDU contains the response data followed by a status code (0x9000 in this case, indicating success).

Once you have the request APDU mapped out, you can start trying to interpret the data. Look for things like TLV (type, length, value) encodings, etc. Look for values like 0x20, since that is a common length (32 bytes) in these sorts of protocols.

The actual algorithm could be either ECDSA, EdDSA, or even ECDH. Tesla key cards, for example, use ECDH. But, as I mentioned earlier, it seems likely that ECDSA is being used here.

Conceptually, the request is going to have some sort of structure identity and some sort of challenge. The key then responds with with a challenge response and information about the identity of the key.

Another approach is to compare the requests/responses across multiple attempts, see which bytes are different and which are the same. Those are likely TLVs, commands, etc.

We can see "41 04 ...." in the response. That 0x41 could be a length.