Plaster of Paris
I'd use a flexible filler. Get a chuck of metal a little wider than the barrel of the key, coat with vaseline or similar and it should pull out when the filler is set.
I think having the lock so far back though would interfere with the window mechanisms etc
It was a clever idea though and is worth someone trying out if they have deep doors.