ListView's header view destroyed on configuration changed

view story

http://stackoverflow.com – I have a ListFragment that uses a header view. Both the header's contents and the list's are fetched from a background task. In order to not re-fetch the data on configuration changes, I am calling setRetainInstance and keeping the data on the fragment. When the the configuration changes, the view is recreated, so it removes the header view that I previously populated. Since now I already have the data, I should just re-add the header view to the list. Unfortunately when I try doing this... boom! java.lang.IllegalStateException: Cannot add header view to list -- setAdapter has already been (HowTos)