I'm getting following error trying to convert Kung Fu Panda 2 from my HD (previously copied by DVDFab) to H.264 mkv file.
Conversion: two pass - first pass: 20:44:55
CUDA GPU acceleration for video decoding enabled
: 20:44:55
CUDA GPU acceleration for video encoding enabled
: 20:44:56
Source( 00372.m2ts )
: 20:44:59
Conversion: two pass - second pass: 21:22:34
CUDA GPU acceleration for video decoding enabled
: 21:22:34
CUDA GPU acceleration for video encoding enabled
: 21:22:34
Source( 00372.m2ts )
: 21:22:43
Work 1 failed, Error = 504.00 0 error: filter name(bdre_demux), type(1), id(22), get std_exception(bad allocation)
 
: 21:27:36
Internal Log is as follows:
DVDFab 8.1.2.6 (2011/11/01  19:01:47)
 0m 01.54s: Qt Translator file load success 
 0m 01.68s: option dvd2dvd 1 dvd2mobile 1 bluray2bluray 1 bd2mobile 1 bd23d 1 bd2dvd 1 file2mobile 1 filemover 1
 0m 08.90s: detected blu-ray
 0m 08.90s: drive I
 0m 08.90s: blu-ray type 1
 0m 08.90s: volume label KUNG_FU_PANDA_2
 0m 10.18s: detected blu-ray
 0m 10.18s: drive I
 0m 10.18s: blu-ray type 1
 0m 10.18s: volume label KUNG_FU_PANDA_2
 0m 13.46s: aacs 1 bd+ 0
 0m 20.95s: got discinfo
 0m 21.51s: got bdmv
 0m 21.91s: Vendor: ATAPI
 0m 21.91s: Product: iHOS104
 0m 21.91s: Revision: WL0D
 0m 21.91s: Vendor specific: 2010/03/08 12:01
 0m 21.91s: AACS version: 1
 0m 21.91s: Number of concurrent AGIDs: 3
 0m 21.91s: Supports bus encryption: NO
 0m 21.91s: Supports binding nonce generation: NO
 0m 21.91s: Supports Hybrid discs: NO
 0m 21.91s: got agid 0
 0m 22.97s: sent host cert chal
 0m 22.97s: got drive cert chal
 0m 22.97s: BEC 0
 0m 22.97s: DRIVE_VENDOR 0x0128
 0m 24.61s: got drive key
 0m 25.25s: got host key signature
 0m 25.25s: verified drive signature
 0m 25.25s: verified host signature
 0m 27.06s: sent host signature and key point
 0m 27.06s: got bus key
 0m 29.11s: got agid 0
 0m 30.14s: sent host cert chal
 0m 30.15s: got drive cert chal
 0m 30.15s: BEC 0
 0m 30.15s: DRIVE_VENDOR 0x0128
 0m 31.85s: got drive key
 0m 31.85s: got host key signature
 0m 31.85s: verified drive signature
 0m 31.85s: verified host signature
 0m 33.68s: sent host signature and key point
 0m 33.68s: got bus key
 0m 33.68s: got volume id
 0m 33.68s: got volume id mac
 0m 33.68s: volume id is correct
 0m 33.68s: got vid
 0m 33.86s: blu-ray B9BD3C262042E4080CE26B8818786BED28A210C4
 0m 34.58s: cert BEE=0x00 same
 0m 35.14s: D 734A402F
 0m 35.14s: got vuk
 0m 35.45s: got unit key 7
 1m 32.07s: got extended bdinfo
 1m 32.07s: opened blu-ray
 3m 04.40s: Blu-ray Copy: Main Movie
 3m 04.40s: Source: I:\BDMV\
 3m 04.40s: Playlist: 0
 3m 04.40s: Chapters: 1 -> 16
 3m 04.40s: TotalSize: 27076 MB
 3m 04.40s: SourceSize: 27076 MB
 3m 04.40s: RemoveHDAudio: 0
 3m 04.40s: CompressToAC3: 0
 3m 04.40s: OutputDisc: BDMV
 3m 04.40s: OutputTarget: BD50
 3m 04.40s: OutputVideo: 1080p/i
 3m 04.42s: info: enable bd copy(mode=0)
 3m 04.42s: info: streams()
 3m 04.42s: info: blu-ray copy begin (unit[elem[atomic[00372.m2ts]]])
 3m 04.42s: info: processing source(00372.m2ts)
77m 04.55s: got bdmv
77m 04.60s: got extended bdinfo
78m 52.89s: 
------  begin mobile work(1/1)  ------
78m 52.89s: info: create config(5)
78m 52.89s: info: convert profile(mkv.remux)
78m 52.89s: info: bluray playlist(0),angle(0),chapter(1->16),3D(0)
78m 52.89s: info: streams((4113.1)(4352.1)(4353.1)(4354.1)(4608.16)(4609.16)(4610.16)(4611.16)(4612.16))
78m 52.91s: info: subpicture preset(2), mode(2)
78m 52.91s: info: subpicture preset(2), mode(2)
78m 52.91s: info: subpicture preset(2), mode(2)
78m 52.91s: info: subpicture preset(2), mode(2)
78m 52.91s: info: subpicture preset(2), mode(2)
103m 08.52s: 
------  begin mobile work(1/1)  ------
103m 08.52s: info: create config(5)
103m 08.52s: info: convert profile(mkv.h264.audiocopy)
103m 08.52s: info: bluray playlist(0),angle(0),chapter(1->16),3D(0)
103m 08.52s: info: streams((4113.2)(4352.1)(4353.1)(4608.16)(4609.16)(4610.16)(4611.16)(4612.16))
103m 08.56s: info: set output video frame_rate(24000/1001)
103m 08.56s: info: subpicture preset(2), mode(2)
103m 08.56s: info: subpicture preset(2), mode(2)
103m 08.56s: info: subpicture preset(2), mode(2)
103m 08.56s: info: subpicture preset(2), mode(2)
103m 08.56s: info: subpicture preset(2), mode(2)
103m 08.57s: try to use gpu acceleration(1) for video decoding
103m 08.57s: CUDA GPU acceleration for video decoding enabled
103m 09.59s: CUDA GPU acceleration for video encoding enabled
103m 09.59s: info: h264_encode: encode param profile(2) level(51) bitrate(8500)
103m 12.63s: info: processing source(00372.m2ts)
140m 47.29s: try to use gpu acceleration(1) for video decoding
140m 47.29s: CUDA GPU acceleration for video decoding enabled
140m 47.31s: CUDA GPU acceleration for video encoding enabled
140m 47.31s: info: h264_encode: encode param profile(2) level(51) bitrate(8500)
140m 55.33s: subpicture(4608) mode(2)
140m 56.34s: info: processing source(00372.m2ts)
145m 34.98s: error: filter name(bdre_demux), type(1), id(22), get std_exception(bad allocation)
145m 49.23s: ------  mobile work failed(504.00 0 error: filter name(bdre_demux), type(1), id(22), get std_exception(bad allocation)
 
)  ------