hadoop - How to control number of records stored in a part file by pig job or a maprecude job? -


is there way control number of records stored in part file?

thanks.

not (if @ all). number of part files in output determined parallelism of script, , data split non-deterministically part files. way can think of like:

a = foreach output generate 1 num ; b = foreach (group all) generate count(a) totaloutputlines ;  -- store both output , b

then, within python wrapper, use totaloutputlines set parallelism of script python wrapper running, par = number of lines in b / number of lines want per file. hopefully, approximately control number of records per part file.

maybe can close want multistorage splitting output 1 file per value of field use.


Comments

Post a Comment

Popular posts from this blog

assembly - 8086 TASM: Illegal Indexing Mode -

this question has answer here: illegal use of register in indirect addressing 1 answer i writing 8086 assembly program needs compile through tasm v3.1. running error can not seem fix. my data segment has following set purposes of keyboard input: parao label byte maxo db 5 acto db ? datao db 5 dup('$') what trying first character entered, first byte of datao: lea dx, datao mov bl, [dx] however, when attempt compile, error: **error** h5.asm(152) illegal indexing mode line 152 "mov bl, [dx]" any appreciated. if matters, running tasm through dosbox (my laptop running 64-bit win7) google hasn't come helpful answers. can post entirety of code if necessary. pretty sure reason can't use dx register pointer. try instead using [si], [di] or [bx]: lea bx, data0 mov al, [bx]
Read more

Java, LWJGL, OpenGL 1.1, decoding BufferedImage to Bytebuffer and binding to OpenGL across classes -

i've been attempting solve problem without luck. have 4 classes, main class, object class, sprite class , image loader class. i'm attempting load png using image loader , using method found here: http://www.java-gaming.org/topics/bufferedimage-to-lwjgl-texture/25516/msg/220280/view.html#msg220280 convert bytebuffer, , bind opengl. the image in it's own seperate resource folder. should being drawn this: ( http://i.imgur.com/j8gbu4c.png ) (32x32), i'm seeing white box has dimensions of image, not actual texture. if has idea of go fix i'd appreciate it. i'm new opengl , looking avoid using external library in order learn how actual code works. thanks. updated! to following, implemented suggestion provided vallentin, quad taking color of first pixel in image. i've attempted implementing image loader provided c.s here . i'm receiving error: javax.imageio.iioexception: error reading png image data @ com.sun.imageio.plugins.png.pngimagereader.r...
Read more

asp.net - Configuring WCF Services in Code WCF 4.5 -

hi, trying configure wcf using code behind, below code: public static void configure(serviceconfiguration config) { string configpath = configurationmanager.appsettings["wcfconfigdbpath"]; // enable “add service reference” support config.description.behaviors.add(new servicemetadatabehavior { httpgetenabled = true }); // set support http, https, net.tcp, net.pipe if (isenabled(configpath, "enablehttp")) config.enableprotocol(new basichttpbinding()); if (isenabled(configpath, "enablenettcp")) config.enableprotocol(new nettcpbinding()); if (isenabled(configpath, "enablepipe")) config.enableprotocol(new netnamedpipebinding()); } private static bool isenabled(string path, string elementname) { try { string elementvalue = string.empty; bool returnval = false; using (xmltextreader reader = new xmltextreader(path)) { reader.readtofoll...
Read more