Eines vorweg:
Code:
ISR, 1Khz {
...
return diff;
}
Hä???


Nun zum Optimieren:

Code:
static int old;                // alter Zählerstand
int new, dir, diff;
Warum sind das alles 16-Bit-Variablen, wo du doch nur mit 4 Bit rechnest und du im Text sogar explizit die 8-Bit-Variante int8_t erwähnst?


Code:
if (new > old) {
   diff = new - old;
   dir = 1;
}
else {
   diff = old - new;
   dir = -1;
}

if (diff > 8) {
   diff = (16 - diff);
   dir *= -1;
}

diff *= dir;
Das würde ich so schreiben:
Code:
diff = new - old;
if (diff > 8)
    diff -= 16;
if (diff < -8)
    diff += 16;